== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:44:32 (1787384672) start mds service on oleg122-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg122-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg122-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 87 (time 1787384801.50 total 10.05 last 8.66) - open/close 285 (time 1787384811.51 total 20.05 last 19.79) - open/close 502 (time 1787384821.52 total 30.06 last 21.67) - open/close 573 (time 1787384833.60 total 42.14 last 5.88) - open/close 774 (time 1787384843.61 total 52.15 last 20.09) - open/close 1003 (time 1787384853.61 total 62.16 last 22.88) - open/close 1215 (time 1787384863.62 total 72.16 last 21.18) - open/close 1401 (time 1787384875.48 total 84.02 last 15.69) - open/close 1587 (time 1787384886.13 total 94.68 last 17.45) - open/close 1758 (time 1787384896.14 total 104.69 last 17.08) - open/close 1950 (time 1787384906.16 total 114.70 last 19.18) - open/close 2171 (time 1787384916.17 total 124.71 last 22.08) - open/close 2340 (time 1787384926.52 total 135.06 last 16.33) - open/close 2572 (time 1787384938.94 total 147.48 last 18.68) - open/close 2716 (time 1787384949.25 total 157.79 last 13.97) - open/close 2956 (time 1787384959.26 total 167.81 last 23.96) - open/close 3213 (time 1787384970.05 total 178.59 last 23.83) - open/close 3445 (time 1787384980.06 total 188.60 last 23.18) - open/close 3689 (time 1787384990.65 total 199.19 last 23.03) - open/close 3953 (time 1787385001.17 total 209.71 last 25.09) - open/close 4115 (time 1787385011.18 total 219.72 last 16.18) - open/close 4301 (time 1787385021.20 total 229.74 last 18.57) - open/close 4528 (time 1787385031.38 total 239.92 last 22.30) - open/close 4754 (time 1787385041.38 total 249.92 last 22.59) - open/close 4967 (time 1787385051.38 total 259.93 last 21.30) - open/close 5042 (time 1787385061.65 total 270.19 last 7.31) - open/close 5189 (time 1787385072.13 total 280.67 last 14.02) - open/close 5473 (time 1787385083.21 total 291.75 last 25.65) - open/close 5618 (time 1787385093.24 total 301.78 last 14.45) - open/close 5808 (time 1787385104.13 total 312.68 last 17.44) - open/close 5952 (time 1787385114.50 total 323.04 last 13.89) - open/close 6162 (time 1787385125.11 total 333.65 last 19.80) - open/close 6379 (time 1787385135.29 total 343.84 last 21.31) - open/close 6538 (time 1787385145.41 total 353.95 last 15.72) - open/close 6748 (time 1787385157.33 total 365.87 last 17.62) - open/close 6917 (time 1787385167.50 total 376.04 last 16.61) - open/close 7158 (time 1787385177.53 total 386.08 last 24.01) - open/close 7414 (time 1787385187.55 total 396.09 last 25.57) - open/close 7545 (time 1787385197.76 total 406.30 last 12.83) - open/close 7677 (time 1787385207.77 total 416.31 last 13.19) - open/close 7943 (time 1787385217.78 total 426.32 last 26.58) - open/close 8183 (time 1787385227.80 total 436.34 last 23.95) - open/close 8439 (time 1787385237.80 total 446.34 last 25.59) - open/close 8680 (time 1787385247.82 total 456.36 last 24.06) - open/close 8927 (time 1787385258.50 total 467.04 last 23.13) - open/close 9206 (time 1787385268.80 total 477.34 last 27.09) - open/close 9518 (time 1787385278.80 total 487.34 last 31.18) - open/close 9802 (time 1787385289.71 total 498.25 last 26.04) - open/close 9973 (time 1787385299.83 total 508.37 last 16.90) - open/close 10204 (time 1787385309.83 total 518.37 last 23.09) - open/close 10484 (time 1787385319.84 total 528.38 last 27.97) - open/close 10705 (time 1787385330.11 total 538.66 last 21.51) - open/close 11028 (time 1787385340.12 total 548.66 last 32.29) - open/close 11280 (time 1787385350.12 total 558.66 last 25.20) - open/close 11518 (time 1787385360.59 total 569.13 last 22.72) - open/close 11764 (time 1787385370.61 total 579.15 last 24.55) - open/close 11888 (time 1787385380.62 total 589.16 last 12.39) - open/close 12099 (time 1787385390.85 total 599.39 last 20.62) - open/close 12296 (time 1787385400.92 total 609.46 last 19.57) - open/close 12498 (time 1787385411.16 total 619.70 last 19.73) - open/close 12789 (time 1787385422.22 total 630.77 last 26.29) - open/close 13022 (time 1787385433.14 total 641.69 last 21.34) - open/close 13202 (time 1787385443.14 total 651.69 last 18.00) - open/close 13417 (time 1787385453.55 total 662.09 last 20.66) - open/close 13599 (time 1787385464.13 total 672.67 last 17.21) - open/close 13844 (time 1787385474.74 total 683.28 last 23.08) - open/close 14148 (time 1787385487.56 total 696.11 last 23.71) - open/close 14385 (time 1787385497.58 total 706.12 last 23.67) - open/close 14562 (time 1787385507.60 total 716.14 last 17.65) - open/close 14745 (time 1787385517.63 total 726.18 last 18.24) - open/close 14949 (time 1787385527.65 total 736.19 last 20.37) - open/close 15140 (time 1787385537.67 total 746.21 last 19.07) - open/close 15319 (time 1787385547.67 total 756.21 last 17.89) - open/close 15615 (time 1787385557.68 total 766.22 last 29.57) - open/close 15834 (time 1787385568.40 total 776.94 last 20.44) - open/close 16024 (time 1787385578.40 total 786.94 last 18.99) - open/close 16293 (time 1787385588.79 total 797.33 last 25.90) - open/close 16409 (time 1787385598.88 total 807.42 last 11.50) - open/close 16577 (time 1787385611.33 total 819.88 last 13.49) - open/close 16789 (time 1787385621.81 total 830.35 last 20.24) - open/close 17024 (time 1787385631.82 total 840.36 last 23.48) - open/close 17180 (time 1787385642.62 total 851.16 last 14.44) - open/close 17351 (time 1787385652.63 total 861.18 last 17.08) - open/close 17611 (time 1787385663.00 total 871.54 last 25.09) - open/close 17734 (time 1787385675.79 total 884.33 last 9.62) - open/close 17870 (time 1787385686.59 total 895.14 last 12.58) - open/close 18161 (time 1787385697.09 total 905.64 last 27.71) - open/close 18385 (time 1787385707.76 total 916.30 last 21.00) - open/close 18589 (time 1787385717.95 total 926.49 last 20.03) - open/close 18841 (time 1787385727.95 total 936.50 last 25.18) - open/close 19096 (time 1787385738.82 total 947.36 last 23.47) - open/close 19326 (time 1787385749.81 total 958.35 last 20.93) - open/close 19437 (time 1787385759.82 total 968.36 last 11.09) - open/close 19664 (time 1787385769.85 total 978.39 last 22.65) - open/close 19750 (time 1787385782.10 total 990.64 last 7.02) - open/close 20000 (time 1787385787.63 total 996.17 last 45.16) - open/close 20799 (time 1787385797.64 total 1006.18 last 79.83) - open/close 21499 (time 1787385807.64 total 1016.18 last 70.00) - open/close 22177 (time 1787385817.64 total 1026.18 last 67.79) - open/close 22923 (time 1787385827.65 total 1036.19 last 74.54) - open/close 23652 (time 1787385837.66 total 1046.20 last 72.82) - open/close 24441 (time 1787385847.67 total 1056.21 last 78.82) - open/close 25218 (time 1787385857.68 total 1066.22 last 77.67) - open/close 25899 (time 1787385867.70 total 1076.24 last 67.93) - open/close 26565 (time 1787385877.71 total 1086.25 last 66.54) - open/close 27264 (time 1787385887.73 total 1096.27 last 69.80) - open/close 28100 (time 1787385897.73 total 1106.27 last 83.58) - open/close 28908 (time 1787385907.74 total 1116.28 last 80.68) - open/close 29830 (time 1787385917.75 total 1126.29 last 92.18) - open/close 30702 (time 1787385927.76 total 1136.30 last 87.10) - open/close 31504 (time 1787385937.78 total 1146.32 last 80.04) - open/close 32199 (time 1787385947.78 total 1156.33 last 69.45) - open/close 32878 (time 1787385957.78 total 1166.33 last 67.89) - open/close 33499 (time 1787385967.79 total 1176.33 last 62.07) - open/close 34104 (time 1787385977.79 total 1186.33 last 60.47) - open/close 34780 (time 1787385987.80 total 1196.34 last 67.58) - open/close 35544 (time 1787385997.80 total 1206.34 last 76.35) - open/close 36241 (time 1787386007.83 total 1216.38 last 69.48) - open/close 36896 (time 1787386017.84 total 1226.38 last 65.49) - open/close 37660 (time 1787386027.84 total 1236.38 last 76.37) - open/close 38376 (time 1787386037.84 total 1246.38 last 71.59) - open/close 39042 (time 1787386047.85 total 1256.39 last 66.52) - open/close 39826 (time 1787386057.85 total 1266.40 last 78.39) - open/close 40000 (time 1787386060.35 total 1268.89 last 69.84) - open/close 40741 (time 1787386070.35 total 1278.89 last 74.08) - open/close 41238 (time 1787386080.39 total 1288.93 last 49.51) - open/close 41713 (time 1787386090.41 total 1298.95 last 47.40) - open/close 42169 (time 1787386100.51 total 1309.05 last 45.14) - open/close 42713 (time 1787386110.51 total 1319.05 last 54.40) - open/close 43363 (time 1787386120.51 total 1329.05 last 65.00) - open/close 44107 (time 1787386130.52 total 1339.06 last 74.31) - open/close 44929 (time 1787386140.54 total 1349.08 last 82.08) - open/close 45728 (time 1787386150.55 total 1359.09 last 79.83) - open/close 46320 (time 1787386160.55 total 1369.09 last 59.17) - open/close 46939 (time 1787386170.58 total 1379.12 last 61.75) - open/close 47627 (time 1787386180.59 total 1389.13 last 68.70) - open/close 48300 (time 1787386190.60 total 1399.14 last 67.23) - open/close 49026 (time 1787386200.61 total 1409.15 last 72.53) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1418.62 seconds: 34.99 ops/second - unlinked 0 (time 1787386215 ; total 0 ; last 0) - unlinked 10000 (time 1787386361 ; total 146 ; last 146) - unlinked 20000 (time 1787386515 ; total 300 ; last 154) - unlinked 30000 (time 1787386658 ; total 443 ; last 143) - unlinked 40000 (time 1787386814 ; total 599 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 742 seconds: 66.888138 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: modules unloaded. pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2