== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:02:34 (1789128154) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg433-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg433-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1789128283.13 total 10.57 last 22.51) - open/close 400 (time 1789128293.96 total 21.40 last 14.96) - open/close 484 (time 1789128304.15 total 31.59 last 8.24) - open/close 618 (time 1789128315.00 total 42.44 last 12.35) - open/close 876 (time 1789128325.01 total 52.46 last 25.76) - open/close 1066 (time 1789128337.81 total 65.25 last 14.85) - open/close 1301 (time 1789128348.77 total 76.21 last 21.45) - open/close 1448 (time 1789128360.01 total 87.45 last 13.08) - open/close 1647 (time 1789128370.20 total 97.64 last 19.53) - open/close 1863 (time 1789128380.21 total 107.65 last 21.58) - open/close 2094 (time 1789128390.61 total 118.05 last 22.21) - open/close 2287 (time 1789128401.53 total 128.97 last 17.67) - open/close 2482 (time 1789128411.55 total 138.99 last 19.46) - open/close 2626 (time 1789128421.56 total 149.00 last 14.38) - open/close 2842 (time 1789128431.57 total 159.01 last 21.57) - open/close 3089 (time 1789128442.42 total 169.86 last 22.77) - open/close 3330 (time 1789128453.12 total 180.56 last 22.53) - open/close 3563 (time 1789128463.16 total 190.60 last 23.20) - open/close 3776 (time 1789128473.92 total 201.36 last 19.80) - open/close 3994 (time 1789128484.29 total 211.73 last 21.02) - open/close 4220 (time 1789128494.79 total 222.23 last 21.51) - open/close 4408 (time 1789128504.80 total 232.25 last 18.78) - open/close 4582 (time 1789128515.08 total 242.52 last 16.93) - open/close 4839 (time 1789128525.72 total 253.16 last 24.16) - open/close 5142 (time 1789128536.14 total 263.58 last 29.07) - open/close 5396 (time 1789128546.14 total 273.59 last 25.39) - open/close 5576 (time 1789128556.15 total 283.59 last 17.99) - open/close 5806 (time 1789128566.15 total 293.59 last 23.00) - open/close 5987 (time 1789128576.15 total 303.60 last 18.10) - open/close 6216 (time 1789128587.15 total 314.60 last 20.82) - open/close 6462 (time 1789128597.81 total 325.26 last 23.08) - open/close 6645 (time 1789128608.22 total 335.67 last 17.58) - open/close 6898 (time 1789128618.24 total 345.68 last 25.27) - open/close 7120 (time 1789128628.24 total 355.68 last 22.20) - open/close 7306 (time 1789128638.98 total 366.42 last 17.31) - open/close 7597 (time 1789128649.87 total 377.31 last 26.73) - open/close 7838 (time 1789128660.35 total 387.79 last 23.00) - open/close 8040 (time 1789128670.37 total 397.82 last 20.14) - open/close 8296 (time 1789128680.41 total 407.85 last 25.51) - open/close 8391 (time 1789128690.41 total 417.86 last 9.50) - open/close 8553 (time 1789128700.42 total 427.86 last 16.19) - open/close 8804 (time 1789128711.07 total 438.52 last 23.57) - open/close 9074 (time 1789128721.08 total 448.52 last 26.98) - open/close 9284 (time 1789128731.96 total 459.40 last 19.30) - open/close 9508 (time 1789128742.14 total 469.58 last 22.00) - open/close 9795 (time 1789128752.15 total 479.60 last 28.67) - open/close 10000 (time 1789128760.07 total 487.51 last 25.89) - open/close 10126 (time 1789128771.84 total 499.28 last 10.71) - open/close 10307 (time 1789128782.60 total 510.04 last 16.81) - open/close 10544 (time 1789128793.36 total 520.80 last 22.03) - open/close 10745 (time 1789128803.37 total 530.81 last 20.07) - open/close 10893 (time 1789128813.56 total 541.00 last 14.52) - open/close 11162 (time 1789128824.40 total 551.84 last 24.82) - open/close 11390 (time 1789128835.15 total 562.59 last 21.21) - open/close 11502 (time 1789128845.35 total 572.79 last 10.98) - open/close 11634 (time 1789128855.84 total 583.29 last 12.58) - open/close 11828 (time 1789128866.38 total 593.82 last 18.41) - open/close 11920 (time 1789128876.40 total 603.85 last 9.18) - open/close 12182 (time 1789128886.41 total 613.85 last 26.19) - open/close 12381 (time 1789128896.41 total 623.86 last 19.89) - open/close 12563 (time 1789128906.52 total 633.96 last 18.01) - open/close 12843 (time 1789128916.93 total 644.37 last 26.89) - open/close 13102 (time 1789128926.93 total 654.38 last 25.89) - open/close 13311 (time 1789128937.23 total 664.68 last 20.29) - open/close 13502 (time 1789128947.24 total 674.68 last 19.09) - open/close 13678 (time 1789128958.22 total 685.66 last 16.03) - open/close 13866 (time 1789128968.47 total 695.91 last 18.34) - open/close 14097 (time 1789128978.48 total 705.93 last 23.07) - open/close 14329 (time 1789128989.24 total 716.68 last 21.57) - open/close 14546 (time 1789128999.26 total 726.70 last 21.66) - open/close 14746 (time 1789129009.28 total 736.72 last 19.95) - open/close 15050 (time 1789129019.29 total 746.74 last 30.36) - open/close 15272 (time 1789129029.76 total 757.20 last 21.21) - open/close 15544 (time 1789129039.76 total 767.21 last 27.19) - open/close 15778 (time 1789129050.37 total 777.81 last 22.07) - open/close 15927 (time 1789129060.38 total 787.82 last 14.88) - open/close 16078 (time 1789129070.69 total 798.14 last 14.64) - open/close 16228 (time 1789129081.48 total 808.93 last 13.90) - open/close 16461 (time 1789129092.18 total 819.62 last 21.78) - open/close 16603 (time 1789129104.13 total 831.58 last 11.88) - open/close 16697 (time 1789129114.16 total 841.60 last 9.38) - open/close 16887 (time 1789129124.39 total 851.83 last 18.58) - open/close 17115 (time 1789129135.00 total 862.44 last 21.49) - open/close 17193 (time 1789129145.01 total 872.45 last 7.79) - open/close 17330 (time 1789129156.01 total 883.45 last 12.46) - open/close 17478 (time 1789129166.03 total 893.48 last 14.76) - open/close 17605 (time 1789129177.10 total 904.54 last 11.48) - open/close 17679 (time 1789129187.34 total 914.78 last 7.23) - open/close 17863 (time 1789129197.37 total 924.81 last 18.34) - open/close 18032 (time 1789129208.35 total 935.80 last 15.39) - open/close 18191 (time 1789129218.36 total 945.80 last 15.89) - open/close 18407 (time 1789129228.86 total 956.30 last 20.57) - open/close 18540 (time 1789129239.38 total 966.83 last 12.64) - open/close 18693 (time 1789129249.99 total 977.43 last 14.43) - open/close 18827 (time 1789129260.23 total 987.67 last 13.08) - open/close 19266 (time 1789129270.24 total 997.68 last 43.86) - open/close 19929 (time 1789129280.24 total 1007.69 last 66.25) - open/close 20629 (time 1789129290.25 total 1017.69 last 69.97) - open/close 21351 (time 1789129300.25 total 1027.70 last 72.17) - open/close 21997 (time 1789129310.27 total 1037.71 last 64.52) - open/close 22707 (time 1789129320.27 total 1047.71 last 70.98) - open/close 23375 (time 1789129330.27 total 1057.72 last 66.77) - open/close 24117 (time 1789129340.28 total 1067.72 last 74.14) - open/close 24821 (time 1789129350.29 total 1077.74 last 70.32) - open/close 25478 (time 1789129360.29 total 1087.74 last 65.70) - open/close 26136 (time 1789129370.30 total 1097.74 last 65.75) - open/close 26808 (time 1789129380.30 total 1107.75 last 67.17) - open/close 27546 (time 1789129390.33 total 1117.77 last 73.62) - open/close 28237 (time 1789129400.34 total 1127.78 last 69.05) - open/close 28875 (time 1789129410.35 total 1137.79 last 63.73) - open/close 29555 (time 1789129420.35 total 1147.79 last 67.98) - open/close 30000 (time 1789129426.77 total 1154.21 last 69.32) - open/close 30635 (time 1789129436.79 total 1164.23 last 63.37) - open/close 31192 (time 1789129446.80 total 1174.24 last 55.64) - open/close 31776 (time 1789129456.83 total 1184.27 last 58.25) - open/close 32459 (time 1789129466.83 total 1194.27 last 68.26) - open/close 33158 (time 1789129476.85 total 1204.29 last 69.81) - open/close 33870 (time 1789129486.85 total 1214.29 last 71.16) - open/close 34480 (time 1789129496.87 total 1224.31 last 60.90) - open/close 35070 (time 1789129506.87 total 1234.32 last 58.96) - open/close 35784 (time 1789129516.88 total 1244.32 last 71.38) - open/close 36488 (time 1789129526.89 total 1254.33 last 70.33) - open/close 37140 (time 1789129536.89 total 1264.33 last 65.19) - open/close 37757 (time 1789129546.90 total 1274.34 last 61.65) - open/close 38469 (time 1789129556.92 total 1284.36 last 71.06) - open/close 39101 (time 1789129566.92 total 1294.37 last 63.15) - open/close 39755 (time 1789129576.93 total 1304.37 last 65.37) - open/close 40000 (time 1789129580.96 total 1308.40 last 60.79) - open/close 40597 (time 1789129590.96 total 1318.40 last 59.69) - open/close 41163 (time 1789129600.97 total 1328.41 last 56.57) - open/close 41824 (time 1789129610.97 total 1338.42 last 66.05) - open/close 42434 (time 1789129620.98 total 1348.43 last 60.94) - open/close 43023 (time 1789129631.00 total 1358.44 last 58.81) - open/close 43761 (time 1789129641.00 total 1368.45 last 73.77) - open/close 44404 (time 1789129651.01 total 1378.45 last 64.27) - open/close 45074 (time 1789129661.02 total 1388.46 last 66.95) - open/close 45821 (time 1789129671.02 total 1398.47 last 74.64) - open/close 46567 (time 1789129681.03 total 1408.47 last 74.56) - open/close 47346 (time 1789129691.03 total 1418.48 last 77.86) - open/close 47898 (time 1789129701.05 total 1428.49 last 55.14) - open/close 48391 (time 1789129711.06 total 1438.51 last 49.21) - open/close 49032 (time 1789129721.07 total 1448.51 last 64.08) - open/close 49591 (time 1789129731.08 total 1458.52 last 55.82) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1459.42 seconds: 34.01 ops/second - unlinked 0 (time 1789129738 ; total 0 ; last 0) - unlinked 10000 (time 1789129916 ; total 178 ; last 178) - unlinked 20000 (time 1789130077 ; total 339 ; last 161) - unlinked 30000 (time 1789130248 ; total 510 ; last 171) - unlinked 40000 (time 1789130410 ; total 672 ; last 162) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 829 seconds: 59.867310 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2