== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:37:58 (1787841478) start mds service on oleg432-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg432-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg432-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1787841600.88 total 10.36 last 17.66) - open/close 326 (time 1787841611.81 total 21.29 last 13.08) - open/close 500 (time 1787841621.81 total 31.29 last 17.40) - open/close 690 (time 1787841632.83 total 42.31 last 17.23) - open/close 795 (time 1787841643.08 total 52.56 last 10.24) - open/close 938 (time 1787841653.29 total 62.77 last 14.01) - open/close 1216 (time 1787841663.30 total 72.78 last 27.78) - open/close 1412 (time 1787841673.30 total 82.78 last 19.59) - open/close 1540 (time 1787841683.39 total 92.87 last 12.69) - open/close 1628 (time 1787841693.73 total 103.21 last 8.51) - open/close 1787 (time 1787841704.24 total 113.72 last 15.14) - open/close 2035 (time 1787841714.24 total 123.72 last 24.80) - open/close 2126 (time 1787841724.32 total 133.80 last 9.02) - open/close 2256 (time 1787841734.42 total 143.91 last 12.87) - open/close 2404 (time 1787841744.74 total 154.22 last 14.34) - open/close 2497 (time 1787841755.00 total 164.48 last 9.07) - open/close 2602 (time 1787841765.81 total 175.29 last 9.71) - open/close 2738 (time 1787841775.81 total 185.30 last 13.59) - open/close 2829 (time 1787841785.82 total 195.30 last 9.10) - open/close 3110 (time 1787841796.58 total 206.07 last 26.09) - open/close 3368 (time 1787841807.40 total 216.88 last 23.86) - open/close 3632 (time 1787841818.21 total 227.69 last 24.42) - open/close 3885 (time 1787841828.22 total 237.71 last 25.26) - open/close 3998 (time 1787841839.02 total 248.50 last 10.46) - open/close 4078 (time 1787841851.11 total 260.59 last 6.62) - open/close 4337 (time 1787841862.22 total 271.70 last 23.32) - open/close 4588 (time 1787841872.23 total 281.71 last 25.07) - open/close 4865 (time 1787841882.25 total 291.73 last 27.64) - open/close 4957 (time 1787841893.41 total 302.89 last 8.25) - open/close 5205 (time 1787841904.22 total 313.70 last 22.94) - open/close 5428 (time 1787841914.23 total 323.71 last 22.28) - open/close 5676 (time 1787841924.86 total 334.34 last 23.33) - open/close 5930 (time 1787841934.88 total 344.36 last 25.36) - open/close 6163 (time 1787841947.68 total 357.17 last 18.20) - open/close 6353 (time 1787841958.44 total 367.92 last 17.67) - open/close 6598 (time 1787841968.44 total 377.92 last 24.49) - open/close 6816 (time 1787841978.81 total 388.29 last 21.01) - open/close 6973 (time 1787841988.82 total 398.30 last 15.70) - open/close 7238 (time 1787841998.83 total 408.31 last 26.47) - open/close 7452 (time 1787842010.02 total 419.50 last 19.12) - open/close 7705 (time 1787842020.77 total 430.25 last 23.52) - open/close 7921 (time 1787842033.33 total 442.81 last 17.21) - open/close 8163 (time 1787842044.03 total 453.51 last 22.61) - open/close 8297 (time 1787842054.04 total 463.52 last 13.40) - open/close 8470 (time 1787842064.80 total 474.28 last 16.07) - open/close 8759 (time 1787842075.00 total 484.48 last 28.33) - open/close 9034 (time 1787842085.76 total 495.24 last 25.56) - open/close 9289 (time 1787842095.77 total 505.25 last 25.47) - open/close 9542 (time 1787842106.34 total 515.82 last 23.94) - open/close 9663 (time 1787842118.07 total 527.55 last 10.32) - open/close 9794 (time 1787842128.07 total 537.55 last 13.09) - open/close 10000 (time 1787842137.23 total 546.71 last 22.51) - open/close 10206 (time 1787842147.23 total 556.71 last 20.60) - open/close 10457 (time 1787842157.54 total 567.02 last 24.34) - open/close 10706 (time 1787842167.54 total 577.02 last 24.89) - open/close 10883 (time 1787842178.53 total 588.02 last 16.10) - open/close 11157 (time 1787842190.67 total 600.15 last 22.58) - open/close 11357 (time 1787842200.96 total 610.44 last 19.43) - open/close 11545 (time 1787842210.98 total 620.46 last 18.77) - open/close 11694 (time 1787842221.57 total 631.05 last 14.07) - open/close 11920 (time 1787842231.58 total 641.07 last 22.57) - open/close 12090 (time 1787842242.56 total 652.04 last 15.49) - open/close 12203 (time 1787842253.60 total 663.09 last 10.23) - open/close 12327 (time 1787842263.77 total 673.25 last 12.20) - open/close 12585 (time 1787842273.91 total 683.40 last 25.43) - open/close 12658 (time 1787842284.95 total 694.43 last 6.61) - open/close 12915 (time 1787842294.97 total 704.45 last 25.67) - open/close 13174 (time 1787842304.98 total 714.46 last 25.86) - open/close 13407 (time 1787842314.99 total 724.47 last 23.28) - open/close 13519 (time 1787842325.01 total 734.49 last 11.18) - open/close 13719 (time 1787842335.40 total 744.88 last 19.25) - open/close 13959 (time 1787842346.86 total 756.35 last 20.93) - open/close 14223 (time 1787842356.86 total 766.35 last 26.40) - open/close 14450 (time 1787842366.89 total 776.37 last 22.65) - open/close 14740 (time 1787842377.00 total 786.48 last 28.68) - open/close 14922 (time 1787842389.48 total 798.96 last 14.59) - open/close 15166 (time 1787842399.48 total 808.96 last 24.38) - open/close 15447 (time 1787842410.03 total 819.51 last 26.64) - open/close 15696 (time 1787842420.04 total 829.52 last 24.87) - open/close 15881 (time 1787842435.62 total 845.10 last 11.87) - open/close 15981 (time 1787842445.73 total 855.21 last 9.89) - open/close 16238 (time 1787842456.43 total 865.91 last 24.02) - open/close 16490 (time 1787842466.73 total 876.21 last 24.47) - open/close 16698 (time 1787842476.74 total 886.22 last 20.78) - open/close 16932 (time 1787842486.74 total 896.22 last 23.39) - open/close 17050 (time 1787842496.74 total 906.23 last 11.80) - open/close 17258 (time 1787842506.79 total 916.27 last 20.71) - open/close 17519 (time 1787842516.82 total 926.30 last 26.02) - open/close 17738 (time 1787842527.72 total 937.20 last 20.10) - open/close 17849 (time 1787842539.45 total 948.93 last 9.46) - open/close 17968 (time 1787842550.20 total 959.68 last 11.07) - open/close 18126 (time 1787842560.88 total 970.37 last 14.78) - open/close 18383 (time 1787842570.89 total 980.37 last 25.70) - open/close 18569 (time 1787842581.03 total 990.51 last 18.34) - open/close 18709 (time 1787842591.32 total 1000.81 last 13.60) - open/close 18917 (time 1787842601.63 total 1011.11 last 20.18) - open/close 19224 (time 1787842611.64 total 1021.12 last 30.67) - open/close 19844 (time 1787842621.65 total 1031.13 last 61.98) - open/close 20000 (time 1787842624.12 total 1033.60 last 63.02) - open/close 20559 (time 1787842634.12 total 1043.60 last 55.89) - open/close 21270 (time 1787842644.13 total 1053.61 last 71.04) - open/close 22061 (time 1787842654.13 total 1063.61 last 79.10) - open/close 22748 (time 1787842664.14 total 1073.62 last 68.62) - open/close 23459 (time 1787842674.15 total 1083.63 last 71.06) - open/close 24076 (time 1787842684.16 total 1093.64 last 61.65) - open/close 24718 (time 1787842694.16 total 1103.64 last 64.16) - open/close 25358 (time 1787842704.17 total 1113.65 last 63.94) - open/close 26042 (time 1787842714.21 total 1123.69 last 68.16) - open/close 26779 (time 1787842724.21 total 1133.69 last 73.67) - open/close 27489 (time 1787842734.23 total 1143.71 last 70.87) - open/close 28243 (time 1787842744.25 total 1153.73 last 75.26) - open/close 28884 (time 1787842754.26 total 1163.74 last 64.03) - open/close 29593 (time 1787842764.26 total 1173.74 last 70.87) - open/close 30000 (time 1787842769.58 total 1179.06 last 76.50) - open/close 30692 (time 1787842779.59 total 1189.07 last 69.13) - open/close 31328 (time 1787842789.60 total 1199.09 last 63.53) - open/close 31950 (time 1787842799.61 total 1209.09 last 62.19) - open/close 32561 (time 1787842809.63 total 1219.11 last 60.97) - open/close 33197 (time 1787842819.63 total 1229.12 last 63.56) - open/close 33960 (time 1787842829.64 total 1239.13 last 76.22) - open/close 34702 (time 1787842839.65 total 1249.13 last 74.14) - open/close 35466 (time 1787842849.66 total 1259.14 last 76.37) - open/close 36179 (time 1787842859.67 total 1269.15 last 71.22) - open/close 36903 (time 1787842869.67 total 1279.15 last 72.37) - open/close 37642 (time 1787842879.68 total 1289.16 last 73.86) - open/close 38413 (time 1787842889.68 total 1299.16 last 77.06) - open/close 39158 (time 1787842899.69 total 1309.17 last 74.45) - open/close 39882 (time 1787842909.70 total 1319.18 last 72.35) - open/close 40692 (time 1787842919.71 total 1329.20 last 80.85) - open/close 41490 (time 1787842929.72 total 1339.20 last 79.74) - open/close 42184 (time 1787842939.72 total 1349.21 last 69.38) - open/close 42821 (time 1787842949.73 total 1359.21 last 63.67) - open/close 43531 (time 1787842959.74 total 1369.22 last 70.93) - open/close 44158 (time 1787842969.75 total 1379.23 last 62.63) - open/close 44776 (time 1787842979.76 total 1389.24 last 61.76) - open/close 45291 (time 1787842989.79 total 1399.27 last 51.34) - open/close 45884 (time 1787842999.80 total 1409.28 last 59.24) - open/close 46561 (time 1787843009.80 total 1419.29 last 67.67) - open/close 47258 (time 1787843019.81 total 1429.29 last 69.69) - open/close 47995 (time 1787843029.81 total 1439.29 last 73.65) - open/close 48696 (time 1787843039.83 total 1449.31 last 69.95) - open/close 49376 (time 1787843049.83 total 1459.32 last 67.99) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1462.75 seconds: 33.93 ops/second - unlinked 0 (time 1787843057 ; total 0 ; last 0) - unlinked 10000 (time 1787843208 ; total 151 ; last 151) - unlinked 20000 (time 1787843359 ; total 302 ; last 151) - unlinked 30000 (time 1787843513 ; total 456 ; last 154) - unlinked 40000 (time 1787843664 ; total 607 ; last 151) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 748 seconds: 66.351601 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: modules unloaded. pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2