== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:06:14 (1786309574) start mds service on oleg125-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: MODOPTS_LIBCFS= oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg125-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 198 (time 1786309702.70 total 10.01 last 19.78) - open/close 389 (time 1786309713.09 total 20.40 last 18.38) - open/close 644 (time 1786309723.10 total 30.40 last 25.49) - open/close 838 (time 1786309733.50 total 40.81 last 18.64) - open/close 1085 (time 1786309743.51 total 50.82 last 24.67) - open/close 1329 (time 1786309753.53 total 60.84 last 24.36) - open/close 1452 (time 1786309763.54 total 70.85 last 12.29) - open/close 1646 (time 1786309773.54 total 80.85 last 19.39) - open/close 1903 (time 1786309784.50 total 91.81 last 23.45) - open/close 2141 (time 1786309794.51 total 101.82 last 23.78) - open/close 2324 (time 1786309804.52 total 111.82 last 18.29) - open/close 2615 (time 1786309814.54 total 121.84 last 29.04) - open/close 2832 (time 1786309824.92 total 132.22 last 20.91) - open/close 2885 (time 1786309835.53 total 142.84 last 4.99) - open/close 3000 (time 1786309846.28 total 153.59 last 10.70) - open/close 3099 (time 1786309857.10 total 164.41 last 9.15) - open/close 3333 (time 1786309867.59 total 174.90 last 22.31) - open/close 3518 (time 1786309877.61 total 184.91 last 18.46) - open/close 3674 (time 1786309887.93 total 195.24 last 15.11) - open/close 3888 (time 1786309897.94 total 205.25 last 21.39) - open/close 4152 (time 1786309908.37 total 215.67 last 25.32) - open/close 4250 (time 1786309921.31 total 228.62 last 7.57) - open/close 4428 (time 1786309932.30 total 239.61 last 16.21) - open/close 4541 (time 1786309942.30 total 249.61 last 11.30) - open/close 4682 (time 1786309952.34 total 259.65 last 14.04) - open/close 4756 (time 1786309962.56 total 269.87 last 7.24) - open/close 4930 (time 1786309973.53 total 280.83 last 15.87) - open/close 5118 (time 1786309983.53 total 290.84 last 18.79) - open/close 5337 (time 1786309994.88 total 302.19 last 19.30) - open/close 5560 (time 1786310004.89 total 312.19 last 22.29) - open/close 5747 (time 1786310015.68 total 322.98 last 17.33) - open/close 5910 (time 1786310025.92 total 333.23 last 15.91) - open/close 6231 (time 1786310036.00 total 343.31 last 31.84) - open/close 6387 (time 1786310046.54 total 353.85 last 14.80) - open/close 6486 (time 1786310058.74 total 366.05 last 8.12) - open/close 6614 (time 1786310068.75 total 376.06 last 12.78) - open/close 6757 (time 1786310078.78 total 386.08 last 14.27) - open/close 6924 (time 1786310089.19 total 396.50 last 16.04) - open/close 7024 (time 1786310100.00 total 407.31 last 9.25) - open/close 7163 (time 1786310113.44 total 420.75 last 10.34) - open/close 7268 (time 1786310124.24 total 431.55 last 9.73) - open/close 7483 (time 1786310134.40 total 441.71 last 21.15) - open/close 7649 (time 1786310145.42 total 452.73 last 15.07) - open/close 7767 (time 1786310156.20 total 463.51 last 10.95) - open/close 7917 (time 1786310167.02 total 474.33 last 13.87) - open/close 8081 (time 1786310177.69 total 485.00 last 15.37) - open/close 8203 (time 1786310187.70 total 495.01 last 12.19) - open/close 8449 (time 1786310197.73 total 505.04 last 24.53) - open/close 8603 (time 1786310207.78 total 515.08 last 15.33) - open/close 8757 (time 1786310219.29 total 526.60 last 13.37) - open/close 8930 (time 1786310230.30 total 537.60 last 15.72) - open/close 9076 (time 1786310241.35 total 548.65 last 13.21) - open/close 9271 (time 1786310251.41 total 558.72 last 19.38) - open/close 9524 (time 1786310261.42 total 568.72 last 25.28) - open/close 9653 (time 1786310272.21 total 579.52 last 11.95) - open/close 9873 (time 1786310283.87 total 591.17 last 18.88) - open/close 10000 (time 1786310288.16 total 595.47 last 29.59) - open/close 10156 (time 1786310299.16 total 606.47 last 14.18) - open/close 10288 (time 1786310309.78 total 617.08 last 12.43) - open/close 10413 (time 1786310319.86 total 627.16 last 12.40) - open/close 10590 (time 1786310329.86 total 637.17 last 17.69) - open/close 10652 (time 1786310339.88 total 647.19 last 6.19) - open/close 10883 (time 1786310349.96 total 657.27 last 22.92) - open/close 11083 (time 1786310360.58 total 667.89 last 18.83) - open/close 11364 (time 1786310371.46 total 678.76 last 25.83) - open/close 11579 (time 1786310382.22 total 689.53 last 19.98) - open/close 11785 (time 1786310392.95 total 700.26 last 19.19) - open/close 11971 (time 1786310402.97 total 710.28 last 18.57) - open/close 12193 (time 1786310413.83 total 721.14 last 20.44) - open/close 12409 (time 1786310424.71 total 732.02 last 19.85) - open/close 12499 (time 1786310434.78 total 742.09 last 8.94) - open/close 12642 (time 1786310445.27 total 752.57 last 13.64) - open/close 12827 (time 1786310455.32 total 762.63 last 18.40) - open/close 12944 (time 1786310468.16 total 775.47 last 9.11) - open/close 13048 (time 1786310478.16 total 785.47 last 10.40) - open/close 13283 (time 1786310488.18 total 795.49 last 23.45) - open/close 13523 (time 1786310498.32 total 805.63 last 23.66) - open/close 13815 (time 1786310509.01 total 816.32 last 27.31) - open/close 14018 (time 1786310519.03 total 826.33 last 20.28) - open/close 14287 (time 1786310529.04 total 836.35 last 26.85) - open/close 14539 (time 1786310540.06 total 847.36 last 22.88) - open/close 14717 (time 1786310550.07 total 857.38 last 17.77) - open/close 14919 (time 1786310561.30 total 868.61 last 18.00) - open/close 15169 (time 1786310572.36 total 879.67 last 22.60) - open/close 15447 (time 1786310582.36 total 889.67 last 27.80) - open/close 15636 (time 1786310592.37 total 899.68 last 18.88) - open/close 15827 (time 1786310602.39 total 909.70 last 19.05) - open/close 16000 (time 1786310612.87 total 920.18 last 16.51) - open/close 16243 (time 1786310627.99 total 935.30 last 16.07) - open/close 16484 (time 1786310638.01 total 945.32 last 24.06) - open/close 16655 (time 1786310650.08 total 957.39 last 14.17) - open/close 16778 (time 1786310661.05 total 968.36 last 11.21) - open/close 16888 (time 1786310671.23 total 978.54 last 10.81) - open/close 16976 (time 1786310682.32 total 989.63 last 7.93) - open/close 17075 (time 1786310692.95 total 1000.25 last 9.32) - open/close 17201 (time 1786310704.02 total 1011.33 last 11.38) - open/close 17386 (time 1786310714.89 total 1022.19 last 17.02) - open/close 17537 (time 1786310725.00 total 1032.31 last 14.92) - open/close 17684 (time 1786310735.57 total 1042.88 last 13.91) - open/close 18120 (time 1786310745.59 total 1052.89 last 43.55) - open/close 18829 (time 1786310755.59 total 1062.90 last 70.86) - open/close 19564 (time 1786310765.59 total 1072.90 last 73.48) - open/close 20000 (time 1786310771.94 total 1079.25 last 68.68) - open/close 20789 (time 1786310781.95 total 1089.26 last 78.84) - open/close 21333 (time 1786310791.96 total 1099.27 last 54.34) - open/close 21881 (time 1786310801.96 total 1109.27 last 54.80) - open/close 22570 (time 1786310811.97 total 1119.27 last 68.85) - open/close 23289 (time 1786310821.98 total 1129.29 last 71.77) - open/close 24027 (time 1786310832.00 total 1139.31 last 73.69) - open/close 24906 (time 1786310842.00 total 1149.31 last 87.87) - open/close 25710 (time 1786310852.01 total 1159.31 last 80.37) - open/close 26524 (time 1786310862.01 total 1169.32 last 81.38) - open/close 27198 (time 1786310872.02 total 1179.33 last 67.33) - open/close 27906 (time 1786310882.03 total 1189.34 last 70.75) - open/close 28601 (time 1786310892.06 total 1199.37 last 69.27) - open/close 29388 (time 1786310902.08 total 1209.38 last 78.60) - open/close 30000 (time 1786310910.64 total 1217.95 last 71.47) - open/close 30691 (time 1786310920.66 total 1227.97 last 68.94) - open/close 31393 (time 1786310930.69 total 1238.00 last 69.97) - open/close 32166 (time 1786310940.70 total 1248.01 last 77.24) - open/close 32951 (time 1786310950.73 total 1258.03 last 78.31) - open/close 33786 (time 1786310960.73 total 1268.04 last 83.47) - open/close 34615 (time 1786310970.74 total 1278.05 last 82.83) - open/close 35295 (time 1786310980.75 total 1288.06 last 67.93) - open/close 36007 (time 1786310990.75 total 1298.06 last 71.17) - open/close 36651 (time 1786311000.76 total 1308.07 last 64.33) - open/close 37337 (time 1786311010.78 total 1318.08 last 68.50) - open/close 38044 (time 1786311020.78 total 1328.09 last 70.68) - open/close 38717 (time 1786311030.78 total 1338.09 last 67.29) - open/close 39450 (time 1786311040.78 total 1348.09 last 73.29) - open/close 40000 (time 1786311048.28 total 1355.59 last 73.37) - open/close 40742 (time 1786311058.30 total 1365.60 last 74.07) - open/close 41390 (time 1786311068.31 total 1375.62 last 64.72) - open/close 42148 (time 1786311078.31 total 1385.62 last 75.80) - open/close 42890 (time 1786311088.32 total 1395.63 last 74.12) - open/close 43521 (time 1786311098.33 total 1405.64 last 63.06) - open/close 44125 (time 1786311108.33 total 1415.64 last 60.36) - open/close 44764 (time 1786311118.34 total 1425.64 last 63.89) - open/close 45453 (time 1786311128.34 total 1435.65 last 68.85) - open/close 46126 (time 1786311138.35 total 1445.66 last 67.23) - open/close 46791 (time 1786311148.35 total 1455.66 last 66.50) - open/close 47533 (time 1786311158.37 total 1465.68 last 74.07) - open/close 48148 (time 1786311168.39 total 1475.70 last 61.39) - open/close 48753 (time 1786311178.39 total 1485.70 last 60.49) - open/close 49473 (time 1786311188.39 total 1495.70 last 71.99) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1497.75 seconds: 33.14 ops/second - unlinked 0 (time 1786311196 ; total 0 ; last 0) - unlinked 10000 (time 1786311338 ; total 142 ; last 142) - unlinked 20000 (time 1786311488 ; total 292 ; last 150) - unlinked 30000 (time 1786311636 ; total 440 ; last 148) - unlinked 40000 (time 1786311786 ; total 590 ; last 150) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 728 seconds: 68.174454 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules via unload_modules_local on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local oleg125-server: modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2