== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:17:38 (1786036658) start mds service on oleg251-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: MODOPTS_LIBCFS= oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg251-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg251-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1786036780.30 total 10.01 last 18.28) - open/close 292 (time 1786036795.80 total 25.51 last 7.03) - open/close 466 (time 1786036806.88 total 36.59 last 15.71) - open/close 578 (time 1786036817.24 total 46.95 last 10.81) - open/close 916 (time 1786036827.88 total 57.59 last 31.76) - open/close 1026 (time 1786036838.17 total 67.88 last 10.69) - open/close 1218 (time 1786036848.18 total 77.88 last 19.19) - open/close 1357 (time 1786036859.22 total 88.93 last 12.58) - open/close 1611 (time 1786036872.86 total 102.57 last 18.63) - open/close 1688 (time 1786036882.86 total 112.57 last 7.70) - open/close 1821 (time 1786036893.61 total 123.32 last 12.37) - open/close 2066 (time 1786036904.56 total 134.26 last 22.38) - open/close 2318 (time 1786036917.26 total 146.96 last 19.84) - open/close 2543 (time 1786036927.26 total 156.97 last 22.50) - open/close 2744 (time 1786036937.28 total 166.99 last 20.06) - open/close 2983 (time 1786036947.99 total 177.69 last 22.32) - open/close 3240 (time 1786036958.36 total 188.06 last 24.78) - open/close 3377 (time 1786036968.38 total 198.08 last 13.67) - open/close 3429 (time 1786036978.84 total 208.54 last 4.97) - open/close 3642 (time 1786036989.16 total 218.87 last 20.63) - open/close 3950 (time 1786036999.96 total 229.67 last 28.52) - open/close 4092 (time 1786037009.98 total 239.69 last 14.17) - open/close 4286 (time 1786037020.00 total 249.70 last 19.36) - open/close 4499 (time 1786037030.03 total 259.74 last 21.23) - open/close 4653 (time 1786037040.04 total 269.75 last 15.38) - open/close 4935 (time 1786037050.05 total 279.75 last 28.20) - open/close 5099 (time 1786037060.07 total 289.77 last 16.37) - open/close 5249 (time 1786037070.09 total 299.79 last 14.97) - open/close 5426 (time 1786037080.99 total 310.69 last 16.24) - open/close 5573 (time 1786037093.02 total 322.73 last 12.21) - open/close 5871 (time 1786037103.98 total 333.69 last 27.19) - open/close 6103 (time 1786037114.40 total 344.11 last 22.26) - open/close 6395 (time 1786037127.51 total 357.22 last 22.28) - open/close 6488 (time 1786037137.52 total 367.23 last 9.29) - open/close 6602 (time 1786037147.53 total 377.23 last 11.39) - open/close 6804 (time 1786037157.53 total 387.24 last 20.18) - open/close 6984 (time 1786037167.60 total 397.31 last 17.88) - open/close 7092 (time 1786037178.22 total 407.93 last 10.17) - open/close 7210 (time 1786037189.82 total 419.52 last 10.18) - open/close 7354 (time 1786037200.68 total 430.39 last 13.26) - open/close 7610 (time 1786037211.37 total 441.08 last 23.94) - open/close 7850 (time 1786037222.43 total 452.13 last 21.71) - open/close 8139 (time 1786037233.48 total 463.19 last 26.14) - open/close 8323 (time 1786037245.86 total 475.57 last 14.86) - open/close 8553 (time 1786037255.87 total 485.58 last 22.97) - open/close 8677 (time 1786037266.58 total 496.28 last 11.59) - open/close 8809 (time 1786037277.41 total 507.11 last 12.19) - open/close 8918 (time 1786037287.82 total 517.53 last 10.47) - open/close 9183 (time 1786037297.83 total 527.53 last 26.48) - open/close 9462 (time 1786037308.82 total 538.53 last 25.37) - open/close 9758 (time 1786037318.84 total 548.55 last 29.55) - open/close 9955 (time 1786037329.74 total 559.45 last 18.07) - open/close 10193 (time 1786037340.64 total 570.35 last 21.84) - open/close 10460 (time 1786037350.64 total 580.35 last 26.70) - open/close 10652 (time 1786037360.65 total 590.36 last 19.18) - open/close 10855 (time 1786037370.66 total 600.36 last 20.29) - open/close 11019 (time 1786037381.15 total 610.85 last 15.64) - open/close 11192 (time 1786037391.20 total 620.90 last 17.21) - open/close 11418 (time 1786037401.97 total 631.67 last 20.99) - open/close 11572 (time 1786037412.09 total 641.79 last 15.21) - open/close 11763 (time 1786037422.10 total 651.81 last 19.07) - open/close 11917 (time 1786037432.24 total 661.95 last 15.20) - open/close 12017 (time 1786037442.39 total 672.09 last 9.85) - open/close 12146 (time 1786037453.35 total 683.06 last 11.77) - open/close 12275 (time 1786037463.92 total 693.63 last 12.20) - open/close 12461 (time 1786037475.93 total 705.63 last 15.49) - open/close 12541 (time 1786037486.15 total 715.85 last 7.83) - open/close 12624 (time 1786037498.60 total 728.31 last 6.67) - open/close 12792 (time 1786037508.60 total 738.31 last 16.80) - open/close 13019 (time 1786037518.61 total 748.31 last 22.68) - open/close 13119 (time 1786037530.01 total 759.71 last 8.77) - open/close 13242 (time 1786037540.61 total 770.32 last 11.60) - open/close 13408 (time 1786037551.18 total 780.89 last 15.70) - open/close 13633 (time 1786037561.20 total 790.90 last 22.47) - open/close 13751 (time 1786037572.06 total 801.77 last 10.86) - open/close 13870 (time 1786037582.31 total 812.02 last 11.61) - open/close 14119 (time 1786037593.19 total 822.89 last 22.90) - open/close 14213 (time 1786037603.54 total 833.25 last 9.08) - open/close 14363 (time 1786037614.18 total 843.89 last 14.10) - open/close 14563 (time 1786037624.54 total 854.25 last 19.30) - open/close 14698 (time 1786037634.56 total 864.27 last 13.48) - open/close 14861 (time 1786037645.56 total 875.27 last 14.82) - open/close 14990 (time 1786037655.74 total 885.45 last 12.67) - open/close 15221 (time 1786037666.30 total 896.01 last 21.87) - open/close 15519 (time 1786037676.31 total 906.02 last 29.78) - open/close 15689 (time 1786037686.99 total 916.70 last 15.92) - open/close 15867 (time 1786037698.03 total 927.74 last 16.13) - open/close 16041 (time 1786037708.66 total 938.37 last 16.37) - open/close 16216 (time 1786037719.26 total 948.97 last 16.50) - open/close 16379 (time 1786037729.27 total 958.98 last 16.28) - open/close 16477 (time 1786037739.28 total 968.99 last 9.80) - open/close 16581 (time 1786037749.55 total 979.26 last 10.12) - open/close 16687 (time 1786037759.82 total 989.53 last 10.32) - open/close 16940 (time 1786037769.82 total 999.53 last 25.29) - open/close 17116 (time 1786037779.83 total 1009.53 last 17.59) - open/close 17339 (time 1786037790.61 total 1020.31 last 20.68) - open/close 17504 (time 1786037800.62 total 1030.33 last 16.48) - open/close 17644 (time 1786037811.37 total 1041.07 last 13.03) - open/close 17794 (time 1786037821.38 total 1051.09 last 14.98) - open/close 18217 (time 1786037831.41 total 1061.12 last 42.16) - open/close 18905 (time 1786037841.42 total 1071.12 last 68.78) - open/close 19672 (time 1786037851.42 total 1081.12 last 76.68) - open/close 20000 (time 1786037854.85 total 1084.56 last 95.47) - open/close 20862 (time 1786037864.86 total 1094.56 last 86.16) - open/close 21600 (time 1786037874.86 total 1104.57 last 73.76) - open/close 22625 (time 1786037884.87 total 1114.58 last 102.42) - open/close 23374 (time 1786037894.88 total 1124.59 last 74.82) - open/close 24126 (time 1786037904.89 total 1134.60 last 75.13) - open/close 24854 (time 1786037914.89 total 1144.60 last 72.78) - open/close 25634 (time 1786037924.91 total 1154.62 last 77.86) - open/close 26449 (time 1786037934.92 total 1164.63 last 81.44) - open/close 27145 (time 1786037944.92 total 1174.63 last 69.58) - open/close 27966 (time 1786037954.92 total 1184.63 last 82.09) - open/close 28826 (time 1786037964.94 total 1194.64 last 85.87) - open/close 29649 (time 1786037974.95 total 1204.66 last 82.21) - open/close 30000 (time 1786037980.09 total 1209.80 last 68.24) - open/close 30726 (time 1786037990.10 total 1219.80 last 72.58) - open/close 31527 (time 1786038000.10 total 1229.81 last 80.04) - open/close 32315 (time 1786038010.10 total 1239.81 last 78.78) - open/close 33043 (time 1786038020.11 total 1249.82 last 72.74) - open/close 33801 (time 1786038030.13 total 1259.83 last 75.69) - open/close 34623 (time 1786038040.13 total 1269.84 last 82.17) - open/close 35356 (time 1786038050.13 total 1279.84 last 73.29) - open/close 36169 (time 1786038060.14 total 1289.85 last 81.24) - open/close 36990 (time 1786038070.14 total 1299.85 last 82.06) - open/close 37736 (time 1786038080.15 total 1309.86 last 74.55) - open/close 38510 (time 1786038090.16 total 1319.87 last 77.33) - open/close 39023 (time 1786038100.16 total 1329.87 last 51.28) - open/close 39546 (time 1786038110.17 total 1339.87 last 52.28) - open/close 40000 (time 1786038117.91 total 1347.62 last 58.63) - open/close 40625 (time 1786038127.91 total 1357.62 last 62.49) - open/close 41290 (time 1786038137.92 total 1367.63 last 66.43) - open/close 42219 (time 1786038147.92 total 1377.63 last 92.88) - open/close 43288 (time 1786038157.93 total 1387.63 last 106.85) - open/close 44035 (time 1786038167.94 total 1397.64 last 74.64) - open/close 44953 (time 1786038177.94 total 1407.65 last 91.72) - open/close 45827 (time 1786038187.95 total 1417.65 last 87.37) - open/close 46790 (time 1786038197.95 total 1427.66 last 96.27) - open/close 47676 (time 1786038207.95 total 1437.66 last 88.59) - open/close 48514 (time 1786038217.96 total 1447.67 last 83.75) - open/close 49243 (time 1786038227.97 total 1457.67 last 72.84) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1462.53 seconds: 33.94 ops/second - unlinked 0 (time 1786038237 ; total 0 ; last 0) - unlinked 10000 (time 1786038361 ; total 124 ; last 124) - unlinked 20000 (time 1786038489 ; total 252 ; last 128) - unlinked 30000 (time 1786038627 ; total 390 ; last 138) - unlinked 40000 (time 1786038754 ; total 517 ; last 127) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 638 seconds: 77.791534 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server unloading modules via unload_modules_local on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local oleg251-server: modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2