== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:02:33 (1787796153) start mds service on oleg245-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg245-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1787796251.02 total 10.29 last 13.11) - open/close 277 (time 1787796262.06 total 21.34 last 12.86) - open/close 435 (time 1787796272.98 total 32.26 last 14.47) - open/close 520 (time 1787796283.60 total 42.87 last 8.01) - open/close 619 (time 1787796293.63 total 52.90 last 9.87) - open/close 788 (time 1787796304.08 total 63.36 last 16.17) - open/close 985 (time 1787796314.08 total 73.36 last 19.70) - open/close 1168 (time 1787796324.32 total 83.59 last 17.87) - open/close 1323 (time 1787796334.34 total 93.61 last 15.47) - open/close 1531 (time 1787796344.65 total 103.92 last 20.17) - open/close 1678 (time 1787796355.41 total 114.68 last 13.66) - open/close 1801 (time 1787796365.42 total 124.69 last 12.29) - open/close 1934 (time 1787796376.35 total 135.62 last 12.16) - open/close 2232 (time 1787796387.03 total 146.31 last 27.90) - open/close 2314 (time 1787796397.04 total 156.32 last 8.19) - open/close 2417 (time 1787796407.92 total 167.20 last 9.47) - open/close 2485 (time 1787796418.12 total 177.40 last 6.67) - open/close 2590 (time 1787796428.14 total 187.41 last 10.48) - open/close 2681 (time 1787796438.60 total 197.88 last 8.70) - open/close 2783 (time 1787796448.62 total 207.90 last 10.18) - open/close 2856 (time 1787796458.91 total 218.18 last 7.10) - open/close 3056 (time 1787796468.92 total 228.19 last 19.98) - open/close 3244 (time 1787796478.92 total 238.20 last 18.79) - open/close 3437 (time 1787796489.52 total 248.79 last 18.21) - open/close 3598 (time 1787796499.52 total 258.80 last 16.09) - open/close 3798 (time 1787796510.03 total 269.30 last 19.04) - open/close 3866 (time 1787796521.03 total 280.31 last 6.18) - open/close 3982 (time 1787796531.18 total 290.45 last 11.43) - open/close 4138 (time 1787796541.73 total 301.00 last 14.79) - open/close 4385 (time 1787796551.74 total 311.01 last 24.68) - open/close 4615 (time 1787796561.75 total 321.03 last 22.96) - open/close 4710 (time 1787796571.95 total 331.23 last 9.31) - open/close 4941 (time 1787796582.41 total 341.68 last 22.09) - open/close 5234 (time 1787796592.86 total 352.13 last 28.05) - open/close 5426 (time 1787796603.30 total 362.57 last 18.38) - open/close 5651 (time 1787796614.22 total 373.49 last 20.61) - open/close 5768 (time 1787796624.42 total 383.69 last 11.47) - open/close 5882 (time 1787796634.43 total 393.70 last 11.39) - open/close 5990 (time 1787796645.32 total 404.59 last 9.92) - open/close 6140 (time 1787796656.27 total 415.55 last 13.69) - open/close 6253 (time 1787796667.15 total 426.42 last 10.39) - open/close 6367 (time 1787796677.59 total 436.86 last 10.93) - open/close 6473 (time 1787796688.54 total 447.81 last 9.68) - open/close 6551 (time 1787796698.77 total 458.04 last 7.63) - open/close 6622 (time 1787796708.90 total 468.17 last 7.01) - open/close 6781 (time 1787796719.91 total 479.18 last 14.44) - open/close 6931 (time 1787796729.92 total 489.19 last 14.98) - open/close 7200 (time 1787796739.92 total 499.19 last 26.89) - open/close 7355 (time 1787796750.11 total 509.39 last 15.21) - open/close 7523 (time 1787796760.15 total 519.42 last 16.74) - open/close 7623 (time 1787796770.82 total 530.10 last 9.37) - open/close 7847 (time 1787796780.88 total 540.16 last 22.26) - open/close 7968 (time 1787796791.89 total 551.17 last 10.99) - open/close 8126 (time 1787796801.90 total 561.18 last 15.78) - open/close 8235 (time 1787796812.82 total 572.09 last 9.98) - open/close 8388 (time 1787796822.85 total 582.12 last 15.26) - open/close 8470 (time 1787796833.33 total 592.60 last 7.83) - open/close 8626 (time 1787796843.67 total 602.94 last 15.08) - open/close 8701 (time 1787796854.75 total 614.02 last 6.77) - open/close 8778 (time 1787796865.99 total 625.26 last 6.85) - open/close 8939 (time 1787796876.03 total 635.31 last 16.03) - open/close 9117 (time 1787796886.93 total 646.20 last 16.34) - open/close 9335 (time 1787796897.76 total 657.03 last 20.13) - open/close 9498 (time 1787796911.12 total 670.40 last 12.19) - open/close 9707 (time 1787796921.60 total 680.87 last 19.95) - open/close 9902 (time 1787796931.62 total 690.89 last 19.46) - open/close 10000 (time 1787796935.75 total 695.02 last 23.72) - open/close 10152 (time 1787796946.14 total 705.42 last 14.63) - open/close 10277 (time 1787796956.17 total 715.45 last 12.46) - open/close 10361 (time 1787796967.06 total 726.33 last 7.72) - open/close 10490 (time 1787796977.07 total 736.34 last 12.89) - open/close 10676 (time 1787796987.55 total 746.83 last 17.73) - open/close 10749 (time 1787796997.56 total 756.83 last 7.30) - open/close 11002 (time 1787797008.54 total 767.81 last 23.04) - open/close 11127 (time 1787797018.63 total 777.90 last 12.39) - open/close 11280 (time 1787797028.73 total 788.00 last 15.15) - open/close 11365 (time 1787797039.40 total 798.67 last 7.97) - open/close 11540 (time 1787797050.05 total 809.33 last 16.42) - open/close 11726 (time 1787797060.69 total 819.96 last 17.49) - open/close 11930 (time 1787797071.57 total 830.84 last 18.76) - open/close 12102 (time 1787797081.58 total 840.85 last 17.19) - open/close 12187 (time 1787797091.60 total 850.87 last 8.48) - open/close 12409 (time 1787797101.61 total 860.89 last 22.17) - open/close 12526 (time 1787797111.66 total 870.93 last 11.64) - open/close 12623 (time 1787797124.32 total 883.59 last 7.67) - open/close 12821 (time 1787797134.32 total 893.59 last 19.79) - open/close 13014 (time 1787797144.33 total 903.60 last 19.28) - open/close 13142 (time 1787797154.37 total 913.64 last 12.75) - open/close 13283 (time 1787797165.61 total 924.89 last 12.54) - open/close 13397 (time 1787797175.62 total 934.89 last 11.39) - open/close 13519 (time 1787797186.46 total 945.73 last 11.25) - open/close 13750 (time 1787797198.28 total 957.55 last 19.55) - open/close 13882 (time 1787797208.85 total 968.12 last 12.48) - open/close 14087 (time 1787797219.77 total 979.04 last 18.78) - open/close 14188 (time 1787797230.11 total 989.38 last 9.77) - open/close 14399 (time 1787797242.71 total 1001.98 last 16.74) - open/close 14582 (time 1787797253.69 total 1012.96 last 16.67) - open/close 14748 (time 1787797263.69 total 1022.96 last 16.60) - open/close 14951 (time 1787797274.52 total 1033.79 last 18.75) - open/close 15064 (time 1787797284.57 total 1043.84 last 11.24) - open/close 15170 (time 1787797299.59 total 1058.86 last 7.06) - open/close 15401 (time 1787797309.60 total 1068.87 last 23.07) - open/close 15644 (time 1787797322.46 total 1081.73 last 18.90) - open/close 15820 (time 1787797333.34 total 1092.61 last 16.17) - open/close 15938 (time 1787797343.71 total 1102.98 last 11.38) - open/close 16495 (time 1787797353.72 total 1112.99 last 55.65) - open/close 17339 (time 1787797363.72 total 1122.99 last 84.37) - open/close 18154 (time 1787797373.72 total 1132.99 last 81.50) - open/close 18978 (time 1787797383.72 total 1142.99 last 82.40) - open/close 19835 (time 1787797393.73 total 1153.00 last 85.59) - open/close 20000 (time 1787797395.78 total 1155.06 last 80.37) - open/close 20844 (time 1787797405.79 total 1165.06 last 84.35) - open/close 21496 (time 1787797415.79 total 1175.07 last 65.18) - open/close 22235 (time 1787797425.83 total 1185.11 last 73.60) - open/close 23003 (time 1787797435.84 total 1195.11 last 76.77) - open/close 23741 (time 1787797445.84 total 1205.12 last 73.75) - open/close 24529 (time 1787797455.86 total 1215.14 last 78.65) - open/close 25293 (time 1787797465.87 total 1225.14 last 76.33) - open/close 25990 (time 1787797475.88 total 1235.15 last 69.63) - open/close 26552 (time 1787797485.89 total 1245.16 last 56.18) - open/close 27180 (time 1787797495.89 total 1255.16 last 62.79) - open/close 27909 (time 1787797505.89 total 1265.16 last 72.89) - open/close 28761 (time 1787797515.90 total 1275.17 last 85.12) - open/close 29506 (time 1787797525.91 total 1285.18 last 74.45) - open/close 30000 (time 1787797532.31 total 1291.59 last 77.08) - open/close 30778 (time 1787797542.32 total 1301.59 last 77.76) - open/close 31535 (time 1787797552.33 total 1311.60 last 75.65) - open/close 32359 (time 1787797562.33 total 1321.60 last 82.35) - open/close 33098 (time 1787797572.35 total 1331.62 last 73.80) - open/close 33848 (time 1787797582.35 total 1341.62 last 74.96) - open/close 34527 (time 1787797592.35 total 1351.62 last 67.90) - open/close 35203 (time 1787797602.35 total 1361.63 last 67.58) - open/close 35987 (time 1787797612.36 total 1371.64 last 78.33) - open/close 36667 (time 1787797622.38 total 1381.66 last 67.86) - open/close 37554 (time 1787797632.39 total 1391.66 last 88.63) - open/close 38254 (time 1787797642.40 total 1401.67 last 69.95) - open/close 38884 (time 1787797652.42 total 1411.70 last 62.84) - open/close 39547 (time 1787797662.43 total 1421.70 last 66.26) - open/close 40000 (time 1787797668.23 total 1427.51 last 78.05) - open/close 40790 (time 1787797678.24 total 1437.51 last 78.94) - open/close 41664 (time 1787797688.25 total 1447.52 last 87.30) - open/close 42475 (time 1787797698.26 total 1457.53 last 81.03) - open/close 43040 (time 1787797708.27 total 1467.54 last 56.45) - open/close 43600 (time 1787797718.27 total 1477.55 last 55.97) - open/close 44405 (time 1787797728.28 total 1487.55 last 80.48) - open/close 45091 (time 1787797738.28 total 1497.56 last 68.56) - open/close 45708 (time 1787797748.28 total 1507.56 last 61.70) - open/close 46351 (time 1787797758.28 total 1517.56 last 64.29) - open/close 47083 (time 1787797768.30 total 1527.57 last 73.09) - open/close 47832 (time 1787797778.30 total 1537.57 last 74.88) - open/close 48609 (time 1787797788.31 total 1547.58 last 77.67) - open/close 49412 (time 1787797798.31 total 1557.58 last 80.27) total: 50000 open/close in 1564.74 seconds: 31.95 ops/second - unlinked 0 (time 1787797811 ; total 0 ; last 0) - unlinked 10000 (time 1787797947 ; total 136 ; last 136) - unlinked 20000 (time 1787798078 ; total 267 ; last 131) - unlinked 30000 (time 1787798205 ; total 394 ; last 127) - unlinked 40000 (time 1787798334 ; total 523 ; last 129) total: 50000 unlinks in 658 seconds: 75.987839 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2