== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:40:21 (1781232021) start mds service on oleg431-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg431-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 66 (time 1781232122.06 total 10.01 last 6.59) - open/close 267 (time 1781232132.14 total 20.10 last 19.93) - open/close 526 (time 1781232143.17 total 31.13 last 23.48) - open/close 799 (time 1781232153.19 total 41.14 last 27.26) - open/close 917 (time 1781232163.71 total 51.67 last 11.21) - open/close 1070 (time 1781232175.38 total 63.33 last 13.12) - open/close 1260 (time 1781232186.19 total 74.15 last 17.56) - open/close 1524 (time 1781232196.20 total 84.15 last 26.38) - open/close 1688 (time 1781232207.13 total 95.09 last 15.00) - open/close 1927 (time 1781232217.14 total 105.10 last 23.88) - open/close 2104 (time 1781232230.09 total 118.05 last 13.67) - open/close 2295 (time 1781232240.10 total 128.05 last 19.09) - open/close 2467 (time 1781232250.90 total 138.86 last 15.92) - open/close 2733 (time 1781232261.83 total 149.78 last 24.35) - open/close 2873 (time 1781232272.41 total 160.36 last 13.24) - open/close 3085 (time 1781232282.56 total 170.51 last 20.88) - open/close 3142 (time 1781232293.25 total 181.21 last 5.33) - open/close 3303 (time 1781232303.27 total 191.23 last 16.07) - open/close 3516 (time 1781232313.42 total 201.37 last 20.99) - open/close 3692 (time 1781232323.54 total 211.49 last 17.39) - open/close 3833 (time 1781232333.55 total 221.51 last 14.08) - open/close 4029 (time 1781232344.29 total 232.24 last 18.26) - open/close 4305 (time 1781232354.30 total 242.25 last 27.57) - open/close 4426 (time 1781232364.32 total 252.27 last 12.08) - open/close 4544 (time 1781232375.04 total 262.99 last 11.00) - open/close 4697 (time 1781232385.73 total 273.68 last 14.31) - open/close 4822 (time 1781232395.74 total 283.69 last 12.49) - open/close 4912 (time 1781232406.10 total 294.05 last 8.68) - open/close 5116 (time 1781232417.17 total 305.12 last 18.43) - open/close 5352 (time 1781232427.18 total 315.13 last 23.57) - open/close 5512 (time 1781232437.91 total 325.87 last 14.91) - open/close 5758 (time 1781232450.92 total 338.87 last 18.92) - open/close 5814 (time 1781232461.56 total 349.52 last 5.26) - open/close 5970 (time 1781232473.91 total 361.87 last 12.63) - open/close 6131 (time 1781232483.94 total 371.89 last 16.06) - open/close 6320 (time 1781232494.74 total 382.69 last 17.50) - open/close 6566 (time 1781232504.96 total 392.92 last 24.06) - open/close 6820 (time 1781232515.53 total 403.48 last 24.03) - open/close 7005 (time 1781232526.60 total 414.55 last 16.71) - open/close 7124 (time 1781232537.48 total 425.43 last 10.94) - open/close 7305 (time 1781232548.10 total 436.05 last 17.04) - open/close 7557 (time 1781232558.12 total 446.07 last 25.14) - open/close 7811 (time 1781232568.13 total 456.09 last 25.37) - open/close 8023 (time 1781232578.50 total 466.46 last 20.44) - open/close 8287 (time 1781232588.51 total 476.46 last 26.39) - open/close 8463 (time 1781232599.44 total 487.39 last 16.10) - open/close 8648 (time 1781232609.46 total 497.41 last 18.46) - open/close 8764 (time 1781232622.09 total 510.04 last 9.19) - open/close 8952 (time 1781232632.15 total 520.11 last 18.68) - open/close 9066 (time 1781232642.16 total 530.11 last 11.40) - open/close 9154 (time 1781232652.79 total 540.75 last 8.27) - open/close 9339 (time 1781232663.56 total 551.51 last 17.19) - open/close 9547 (time 1781232673.56 total 561.52 last 20.79) - open/close 9809 (time 1781232683.71 total 571.66 last 25.82) - open/close 10000 (time 1781232690.03 total 577.98 last 30.24) - open/close 10120 (time 1781232702.03 total 589.98 last 10.00) - open/close 10370 (time 1781232712.03 total 599.99 last 24.98) - open/close 10499 (time 1781232722.04 total 609.99 last 12.90) - open/close 10687 (time 1781232732.05 total 620.00 last 18.78) - open/close 10908 (time 1781232743.31 total 631.27 last 19.62) - open/close 11103 (time 1781232753.33 total 641.28 last 19.46) - open/close 11359 (time 1781232763.34 total 651.30 last 25.57) - open/close 11471 (time 1781232773.35 total 661.31 last 11.19) - open/close 11608 (time 1781232784.26 total 672.21 last 12.56) - open/close 11699 (time 1781232794.97 total 682.93 last 8.49) - open/close 11904 (time 1781232805.66 total 693.62 last 19.17) - open/close 12058 (time 1781232818.42 total 706.37 last 12.08) - open/close 12233 (time 1781232829.25 total 717.21 last 16.15) - open/close 12396 (time 1781232842.19 total 730.14 last 12.60) - open/close 12656 (time 1781232852.58 total 740.53 last 25.03) - open/close 12777 (time 1781232862.80 total 750.76 last 11.83) - open/close 12891 (time 1781232872.82 total 760.77 last 11.38) - open/close 12981 (time 1781232885.39 total 773.35 last 7.16) - open/close 13131 (time 1781232895.40 total 783.36 last 14.99) - open/close 13247 (time 1781232906.32 total 794.28 last 10.62) - open/close 13402 (time 1781232916.35 total 804.30 last 15.47) - open/close 13557 (time 1781232927.95 total 815.90 last 13.36) - open/close 13634 (time 1781232938.19 total 826.14 last 7.52) - open/close 13809 (time 1781232948.60 total 836.56 last 16.80) - open/close 13896 (time 1781232958.62 total 846.57 last 8.69) - open/close 14033 (time 1781232968.71 total 856.66 last 13.58) - open/close 14221 (time 1781232978.95 total 866.91 last 18.35) - open/close 14388 (time 1781232989.72 total 877.67 last 15.51) - open/close 14592 (time 1781233000.08 total 888.03 last 19.69) - open/close 14749 (time 1781233014.49 total 902.44 last 10.90) - open/close 14934 (time 1781233025.22 total 913.18 last 17.23) - open/close 15075 (time 1781233035.55 total 923.51 last 13.65) - open/close 15259 (time 1781233046.36 total 934.31 last 17.03) - open/close 15407 (time 1781233059.06 total 947.02 last 11.65) - open/close 15545 (time 1781233069.08 total 957.03 last 13.78) - open/close 15633 (time 1781233079.51 total 967.46 last 8.44) - open/close 15800 (time 1781233089.52 total 977.47 last 16.69) - open/close 15886 (time 1781233099.90 total 987.85 last 8.28) - open/close 16106 (time 1781233112.78 total 1000.73 last 17.08) - open/close 16217 (time 1781233123.34 total 1011.30 last 10.51) - open/close 16383 (time 1781233133.89 total 1021.85 last 15.73) - open/close 16612 (time 1781233143.91 total 1031.87 last 22.85) - open/close 16811 (time 1781233153.92 total 1041.87 last 19.89) - open/close 17082 (time 1781233164.05 total 1052.00 last 26.74) - open/close 17296 (time 1781233174.22 total 1062.17 last 21.04) - open/close 17831 (time 1781233184.24 total 1072.20 last 53.38) - open/close 18364 (time 1781233194.24 total 1082.20 last 53.30) - open/close 19023 (time 1781233204.25 total 1092.20 last 65.89) - open/close 19785 (time 1781233214.25 total 1102.21 last 76.15) - open/close 20000 (time 1781233216.88 total 1104.83 last 81.91) - open/close 20800 (time 1781233226.88 total 1114.84 last 79.95) - open/close 21565 (time 1781233236.88 total 1124.84 last 76.50) - open/close 22378 (time 1781233246.90 total 1134.85 last 81.20) - open/close 23157 (time 1781233256.91 total 1144.86 last 77.79) - open/close 23890 (time 1781233266.91 total 1154.87 last 73.28) - open/close 24705 (time 1781233276.92 total 1164.87 last 81.43) - open/close 25483 (time 1781233286.92 total 1174.88 last 77.78) - open/close 26252 (time 1781233296.92 total 1184.88 last 76.89) - open/close 27017 (time 1781233306.93 total 1194.88 last 76.45) - open/close 27789 (time 1781233316.93 total 1204.89 last 77.19) - open/close 28595 (time 1781233326.93 total 1214.89 last 80.60) - open/close 29268 (time 1781233336.94 total 1224.89 last 67.27) - open/close 29961 (time 1781233346.97 total 1234.92 last 69.08) - open/close 30616 (time 1781233357.03 total 1244.98 last 65.11) - open/close 31320 (time 1781233367.05 total 1255.01 last 70.23) - open/close 31998 (time 1781233377.07 total 1265.02 last 67.72) - open/close 32748 (time 1781233387.08 total 1275.03 last 74.91) - open/close 33556 (time 1781233397.08 total 1285.03 last 80.79) - open/close 34342 (time 1781233407.08 total 1295.03 last 78.59) - open/close 35107 (time 1781233417.08 total 1305.04 last 76.48) - open/close 35862 (time 1781233427.08 total 1315.04 last 75.50) - open/close 36556 (time 1781233437.08 total 1325.04 last 69.40) - open/close 37326 (time 1781233447.09 total 1335.04 last 76.94) - open/close 38150 (time 1781233457.10 total 1345.05 last 82.34) - open/close 38911 (time 1781233467.10 total 1355.06 last 76.05) - open/close 39758 (time 1781233477.12 total 1365.07 last 84.60) - open/close 40000 (time 1781233480.48 total 1368.43 last 72.03) - open/close 40711 (time 1781233490.49 total 1378.44 last 71.03) - open/close 41308 (time 1781233500.49 total 1388.44 last 59.69) - open/close 41915 (time 1781233510.49 total 1398.44 last 60.69) - open/close 42480 (time 1781233520.50 total 1408.45 last 56.46) - open/close 43159 (time 1781233530.50 total 1418.45 last 67.88) - open/close 43857 (time 1781233540.51 total 1428.46 last 69.74) - open/close 44532 (time 1781233550.52 total 1438.48 last 67.38) - open/close 45190 (time 1781233560.56 total 1448.51 last 65.57) - open/close 45898 (time 1781233570.59 total 1458.55 last 70.55) - open/close 46558 (time 1781233580.63 total 1468.58 last 65.75) - open/close 47313 (time 1781233590.70 total 1478.66 last 74.94) - open/close 48078 (time 1781233600.71 total 1488.67 last 76.44) - open/close 48935 (time 1781233610.76 total 1498.72 last 85.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1508.50 seconds: 32.90 ops/second - unlinked 0 (time 1781233625 ; total 0 ; last 0) - unlinked 10000 (time 1781233770 ; total 145 ; last 145) - unlinked 20000 (time 1781233913 ; total 288 ; last 143) - unlinked 30000 (time 1781234050 ; total 425 ; last 137) - unlinked 40000 (time 1781234186 ; total 561 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 683 seconds: 72.666176 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2