== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:39:45 (1785317985) start mds service on oleg131-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 116 (time 1785318094.49 total 10.02 last 11.57) - open/close 302 (time 1785318104.62 total 20.15 last 18.36) - open/close 412 (time 1785318115.65 total 31.18 last 9.98) - open/close 571 (time 1785318125.75 total 41.28 last 15.74) - open/close 654 (time 1785318136.43 total 51.96 last 7.77) - open/close 786 (time 1785318146.64 total 62.17 last 12.93) - open/close 910 (time 1785318158.23 total 73.76 last 10.70) - open/close 1123 (time 1785318168.24 total 83.77 last 21.29) - open/close 1328 (time 1785318178.24 total 93.77 last 20.49) - open/close 1474 (time 1785318188.79 total 104.32 last 13.83) - open/close 1636 (time 1785318199.28 total 114.81 last 15.45) - open/close 1728 (time 1785318209.29 total 124.82 last 9.19) - open/close 1861 (time 1785318219.64 total 135.16 last 12.86) - open/close 1949 (time 1785318229.68 total 145.21 last 8.76) - open/close 2038 (time 1785318239.74 total 155.27 last 8.84) - open/close 2139 (time 1785318249.75 total 165.28 last 10.10) - open/close 2282 (time 1785318260.49 total 176.02 last 13.31) - open/close 2419 (time 1785318270.49 total 186.02 last 13.69) - open/close 2568 (time 1785318281.16 total 196.69 last 13.97) - open/close 2722 (time 1785318291.17 total 206.70 last 15.39) - open/close 2891 (time 1785318301.33 total 216.86 last 16.62) - open/close 3088 (time 1785318312.12 total 227.65 last 18.26) - open/close 3274 (time 1785318322.87 total 238.40 last 17.30) - open/close 3344 (time 1785318333.89 total 249.42 last 6.35) - open/close 3521 (time 1785318344.24 total 259.77 last 17.11) - open/close 3591 (time 1785318354.48 total 270.01 last 6.83) - open/close 3668 (time 1785318365.18 total 280.71 last 7.20) - open/close 3859 (time 1785318375.67 total 291.20 last 18.21) - open/close 3968 (time 1785318385.78 total 301.31 last 10.78) - open/close 4106 (time 1785318396.67 total 312.20 last 12.67) - open/close 4205 (time 1785318406.89 total 322.42 last 9.68) - open/close 4286 (time 1785318417.32 total 332.85 last 7.77) - open/close 4398 (time 1785318427.34 total 342.87 last 11.18) - open/close 4462 (time 1785318437.35 total 352.88 last 6.39) - open/close 4617 (time 1785318448.41 total 363.94 last 14.01) - open/close 4764 (time 1785318458.54 total 374.07 last 14.51) - open/close 4886 (time 1785318469.04 total 384.57 last 11.62) - open/close 4996 (time 1785318479.16 total 394.69 last 10.87) - open/close 5043 (time 1785318491.89 total 407.42 last 3.69) - open/close 5151 (time 1785318502.57 total 418.10 last 10.11) - open/close 5220 (time 1785318512.58 total 428.11 last 6.90) - open/close 5362 (time 1785318523.64 total 439.17 last 12.83) - open/close 5537 (time 1785318533.65 total 449.18 last 17.48) - open/close 5680 (time 1785318543.66 total 459.19 last 14.29) - open/close 5773 (time 1785318554.10 total 469.63 last 8.91) - open/close 5919 (time 1785318564.98 total 480.51 last 13.42) - open/close 6065 (time 1785318575.25 total 490.78 last 14.21) - open/close 6247 (time 1785318585.26 total 500.79 last 18.19) - open/close 6480 (time 1785318595.84 total 511.37 last 22.02) - open/close 6602 (time 1785318606.09 total 521.62 last 11.91) - open/close 6719 (time 1785318616.88 total 532.41 last 10.84) - open/close 6809 (time 1785318628.58 total 544.11 last 7.69) - open/close 6909 (time 1785318639.43 total 554.96 last 9.22) - open/close 6983 (time 1785318650.24 total 565.77 last 6.85) - open/close 7073 (time 1785318662.85 total 578.38 last 7.13) - open/close 7190 (time 1785318672.88 total 588.41 last 11.66) - open/close 7291 (time 1785318683.91 total 599.44 last 9.16) - open/close 7397 (time 1785318693.93 total 609.46 last 10.58) - open/close 7458 (time 1785318706.02 total 621.55 last 5.04) - open/close 7721 (time 1785318716.35 total 631.88 last 25.45) - open/close 7878 (time 1785318726.37 total 641.90 last 15.67) - open/close 8045 (time 1785318736.77 total 652.30 last 16.06) - open/close 8174 (time 1785318746.78 total 662.31 last 12.89) - open/close 8325 (time 1785318757.33 total 672.86 last 14.31) - open/close 8428 (time 1785318767.68 total 683.21 last 9.94) - open/close 8577 (time 1785318777.71 total 693.24 last 14.86) - open/close 8681 (time 1785318788.44 total 703.97 last 9.70) - open/close 8875 (time 1785318798.45 total 713.98 last 19.37) - open/close 8988 (time 1785318808.76 total 724.29 last 10.97) - open/close 9110 (time 1785318818.79 total 734.32 last 12.16) - open/close 9220 (time 1785318828.90 total 744.43 last 10.88) - open/close 9291 (time 1785318839.49 total 755.02 last 6.70) - open/close 9424 (time 1785318850.03 total 765.56 last 12.62) - open/close 9615 (time 1785318860.07 total 775.60 last 19.03) - open/close 9763 (time 1785318870.40 total 785.92 last 14.33) - open/close 9910 (time 1785318880.72 total 796.25 last 14.24) - open/close 10000 (time 1785318885.94 total 801.47 last 17.23) - open/close 10105 (time 1785318895.94 total 811.47 last 10.50) - open/close 10302 (time 1785318906.76 total 822.29 last 18.20) - open/close 10384 (time 1785318917.17 total 832.70 last 7.88) - open/close 10612 (time 1785318927.17 total 842.70 last 22.80) - open/close 10802 (time 1785318938.15 total 853.68 last 17.30) - open/close 10942 (time 1785318948.98 total 864.51 last 12.93) - open/close 11086 (time 1785318959.28 total 874.81 last 13.97) - open/close 11185 (time 1785318969.40 total 884.92 last 9.79) - open/close 11284 (time 1785318979.52 total 895.05 last 9.77) - open/close 11455 (time 1785318989.54 total 905.07 last 17.07) - open/close 11536 (time 1785319004.05 total 919.58 last 5.58) - open/close 11662 (time 1785319015.11 total 930.64 last 11.39) - open/close 11742 (time 1785319025.99 total 941.52 last 7.36) - open/close 11931 (time 1785319036.54 total 952.07 last 17.91) - open/close 12022 (time 1785319047.54 total 963.07 last 8.28) - open/close 12121 (time 1785319058.42 total 973.95 last 9.09) - open/close 12253 (time 1785319069.19 total 984.72 last 12.26) - open/close 12451 (time 1785319079.23 total 994.75 last 19.73) - open/close 12562 (time 1785319089.55 total 1005.08 last 10.75) - open/close 12635 (time 1785319099.84 total 1015.37 last 7.09) - open/close 12728 (time 1785319109.86 total 1025.39 last 9.28) - open/close 12836 (time 1785319119.86 total 1035.39 last 10.80) - open/close 12955 (time 1785319129.90 total 1045.43 last 11.85) - open/close 13067 (time 1785319140.53 total 1056.06 last 10.54) - open/close 13207 (time 1785319150.54 total 1066.07 last 13.99) - open/close 13313 (time 1785319160.56 total 1076.08 last 10.58) - open/close 13383 (time 1785319171.24 total 1086.77 last 6.55) - open/close 13499 (time 1785319181.25 total 1096.78 last 11.58) - open/close 13660 (time 1785319192.95 total 1108.48 last 13.76) - open/close 13795 (time 1785319202.97 total 1118.49 last 13.48) - open/close 13872 (time 1785319213.63 total 1129.16 last 7.22) - open/close 13981 (time 1785319223.64 total 1139.17 last 10.89) - open/close 14103 (time 1785319234.25 total 1149.78 last 11.51) - open/close 14253 (time 1785319245.00 total 1160.53 last 13.95) - open/close 14362 (time 1785319255.74 total 1171.27 last 10.15) - open/close 14497 (time 1785319265.75 total 1181.28 last 13.49) - open/close 14578 (time 1785319276.43 total 1191.96 last 7.58) - open/close 14698 (time 1785319287.40 total 1202.93 last 10.93) - open/close 14867 (time 1785319297.56 total 1213.09 last 16.64) - open/close 15005 (time 1785319307.96 total 1223.49 last 13.26) - open/close 15330 (time 1785319317.97 total 1233.50 last 32.49) - open/close 15906 (time 1785319327.97 total 1243.50 last 57.57) - open/close 16448 (time 1785319338.00 total 1253.53 last 54.05) - open/close 16890 (time 1785319348.01 total 1263.54 last 44.14) - open/close 17345 (time 1785319358.02 total 1273.55 last 45.46) - open/close 17708 (time 1785319368.04 total 1283.57 last 36.24) - open/close 18105 (time 1785319378.04 total 1293.57 last 39.67) - open/close 18705 (time 1785319388.07 total 1303.59 last 59.88) - open/close 19237 (time 1785319398.08 total 1313.61 last 53.11) - open/close 19701 (time 1785319408.09 total 1323.62 last 46.37) - open/close 20000 (time 1785319414.05 total 1329.58 last 50.19) - open/close 20530 (time 1785319424.05 total 1339.58 last 52.98) - open/close 21062 (time 1785319434.06 total 1349.59 last 53.16) - open/close 21688 (time 1785319444.07 total 1359.59 last 62.55) - open/close 22233 (time 1785319454.08 total 1369.61 last 54.43) - open/close 22723 (time 1785319464.09 total 1379.62 last 48.95) - open/close 23144 (time 1785319474.10 total 1389.62 last 42.07) - open/close 23649 (time 1785319484.10 total 1399.63 last 50.49) - open/close 24088 (time 1785319494.11 total 1409.64 last 43.85) - open/close 24560 (time 1785319504.11 total 1419.64 last 47.20) - open/close 25135 (time 1785319514.13 total 1429.66 last 57.36) - open/close 25717 (time 1785319524.15 total 1439.68 last 58.11) - open/close 26468 (time 1785319534.15 total 1449.68 last 75.05) - open/close 27243 (time 1785319544.15 total 1459.68 last 77.49) - open/close 27918 (time 1785319554.16 total 1469.69 last 67.49) - open/close 28752 (time 1785319564.19 total 1479.72 last 83.14) - open/close 29324 (time 1785319574.20 total 1489.73 last 57.12) - open/close 29860 (time 1785319584.22 total 1499.75 last 53.48) - open/close 30000 (time 1785319587.30 total 1502.83 last 45.49) - open/close 30531 (time 1785319597.31 total 1512.84 last 53.05) - open/close 31338 (time 1785319607.32 total 1522.85 last 80.66) - open/close 31940 (time 1785319617.36 total 1532.89 last 59.92) - open/close 32485 (time 1785319627.37 total 1542.90 last 54.47) - open/close 33046 (time 1785319637.42 total 1552.95 last 55.79) - open/close 33698 (time 1785319647.44 total 1562.97 last 65.09) - open/close 34210 (time 1785319657.45 total 1572.98 last 51.13) - open/close 34762 (time 1785319667.46 total 1582.99 last 55.17) - open/close 35277 (time 1785319677.46 total 1592.99 last 51.48) - open/close 35800 (time 1785319687.47 total 1603.00 last 52.29) - open/close 36363 (time 1785319697.49 total 1613.02 last 56.18) - open/close 36876 (time 1785319707.52 total 1623.05 last 51.13) - open/close 37511 (time 1785319717.53 total 1633.06 last 63.45) - open/close 38063 (time 1785319727.53 total 1643.06 last 55.20) - open/close 38614 (time 1785319737.54 total 1653.07 last 55.05) - open/close 39555 (time 1785319747.54 total 1663.07 last 94.08) - open/close 40000 (time 1785319752.64 total 1668.17 last 87.32) - open/close 40507 (time 1785319762.64 total 1678.17 last 50.68) - open/close 41100 (time 1785319772.64 total 1688.17 last 59.28) - open/close 41719 (time 1785319782.65 total 1698.18 last 61.86) - open/close 42294 (time 1785319792.66 total 1708.19 last 57.42) - open/close 42845 (time 1785319802.68 total 1718.21 last 54.99) - open/close 43406 (time 1785319812.71 total 1728.24 last 55.96) - open/close 44221 (time 1785319822.71 total 1738.24 last 81.44) - open/close 45014 (time 1785319832.72 total 1748.25 last 79.24) - open/close 45531 (time 1785319842.73 total 1758.26 last 51.65) - open/close 46003 (time 1785319852.74 total 1768.27 last 47.17) - open/close 46769 (time 1785319862.75 total 1778.28 last 76.49) - open/close 47718 (time 1785319872.76 total 1788.29 last 94.81) - open/close 48296 (time 1785319882.78 total 1798.31 last 57.72) - open/close 48834 (time 1785319892.79 total 1808.32 last 53.71) - open/close 49320 (time 1785319902.80 total 1818.33 last 48.56) - open/close 49853 (time 1785319912.81 total 1828.34 last 53.26) total: 50000 open/close in 1830.38 seconds: 27.32 ops/second - unlinked 0 (time 1785319920 ; total 0 ; last 0) - unlinked 10000 (time 1785320094 ; total 174 ; last 174) - unlinked 20000 (time 1785320249 ; total 329 ; last 155) - unlinked 30000 (time 1785320412 ; total 492 ; last 163) - unlinked 40000 (time 1785320578 ; total 658 ; last 166) total: 50000 unlinks in 815 seconds: 61.349693 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2