== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:28:36 (1775197716) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg358-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 260 (time 1775197817.36 total 10.91 last 23.84) - open/close 465 (time 1775197828.31 total 21.86 last 18.71) - open/close 620 (time 1775197841.32 total 34.87 last 11.92) - open/close 831 (time 1775197851.33 total 44.88 last 21.07) - open/close 1056 (time 1775197862.16 total 55.71 last 20.77) - open/close 1195 (time 1775197872.22 total 65.77 last 13.83) - open/close 1459 (time 1775197882.23 total 75.77 last 26.38) - open/close 1667 (time 1775197892.23 total 85.78 last 20.79) - open/close 1907 (time 1775197903.18 total 96.73 last 21.92) - open/close 2158 (time 1775197913.61 total 107.15 last 24.07) - open/close 2396 (time 1775197923.97 total 117.52 last 22.96) - open/close 2618 (time 1775197933.98 total 127.53 last 22.17) - open/close 2896 (time 1775197944.00 total 137.54 last 27.76) - open/close 3146 (time 1775197954.45 total 148.00 last 23.91) - open/close 3385 (time 1775197964.77 total 158.31 last 23.17) - open/close 3621 (time 1775197975.24 total 168.79 last 22.53) - open/close 3877 (time 1775197985.24 total 178.79 last 25.60) - open/close 4092 (time 1775197997.41 total 190.96 last 17.67) - open/close 4306 (time 1775198009.04 total 202.58 last 18.41) - open/close 4491 (time 1775198019.05 total 212.60 last 18.48) - open/close 4704 (time 1775198029.05 total 222.60 last 21.30) - open/close 4922 (time 1775198041.03 total 234.58 last 18.19) - open/close 5112 (time 1775198051.05 total 244.60 last 18.97) - open/close 5362 (time 1775198061.84 total 255.39 last 23.16) - open/close 5614 (time 1775198071.85 total 265.40 last 25.17) - open/close 5819 (time 1775198082.58 total 276.13 last 19.10) - open/close 6102 (time 1775198092.60 total 286.14 last 28.27) - open/close 6335 (time 1775198102.61 total 296.16 last 23.27) - open/close 6570 (time 1775198112.79 total 306.34 last 23.08) - open/close 6829 (time 1775198122.80 total 316.35 last 25.87) - open/close 7057 (time 1775198133.59 total 327.13 last 21.14) - open/close 7194 (time 1775198143.59 total 337.14 last 13.69) - open/close 7326 (time 1775198153.61 total 347.15 last 13.18) - open/close 7459 (time 1775198163.62 total 357.17 last 13.28) - open/close 7692 (time 1775198174.02 total 367.57 last 22.40) - open/close 7963 (time 1775198184.03 total 377.58 last 27.08) - open/close 8069 (time 1775198195.10 total 388.65 last 9.57) - open/close 8163 (time 1775198205.10 total 398.65 last 9.40) - open/close 8254 (time 1775198215.11 total 408.66 last 9.09) - open/close 8332 (time 1775198225.24 total 418.79 last 7.70) - open/close 8535 (time 1775198235.25 total 428.80 last 20.28) - open/close 8751 (time 1775198245.85 total 439.40 last 20.38) - open/close 9016 (time 1775198255.86 total 449.41 last 26.47) - open/close 9281 (time 1775198266.13 total 459.68 last 25.80) - open/close 9508 (time 1775198277.01 total 470.56 last 20.86) - open/close 9702 (time 1775198287.08 total 480.63 last 19.25) - open/close 9885 (time 1775198297.10 total 490.65 last 18.28) - open/close 10000 (time 1775198302.58 total 496.13 last 20.96) - open/close 10249 (time 1775198313.51 total 507.05 last 22.80) - open/close 10503 (time 1775198323.86 total 517.41 last 24.53) - open/close 10685 (time 1775198333.87 total 527.42 last 18.18) - open/close 10857 (time 1775198343.87 total 537.42 last 17.20) - open/close 11060 (time 1775198353.87 total 547.42 last 20.29) - open/close 11306 (time 1775198364.15 total 557.70 last 23.94) - open/close 11379 (time 1775198375.90 total 569.45 last 6.21) - open/close 11592 (time 1775198386.33 total 579.88 last 20.41) - open/close 11812 (time 1775198396.72 total 590.27 last 21.17) - open/close 12040 (time 1775198407.46 total 601.01 last 21.23) - open/close 12243 (time 1775198417.47 total 611.02 last 20.29) - open/close 12447 (time 1775198427.54 total 621.09 last 20.25) - open/close 12649 (time 1775198438.05 total 631.60 last 19.23) - open/close 12872 (time 1775198448.06 total 641.61 last 22.28) - open/close 13033 (time 1775198459.10 total 652.65 last 14.58) - open/close 13112 (time 1775198470.23 total 663.78 last 7.10) - open/close 13305 (time 1775198480.25 total 673.80 last 19.27) - open/close 13525 (time 1775198490.56 total 684.11 last 21.33) - open/close 13758 (time 1775198501.61 total 695.16 last 21.09) - open/close 13904 (time 1775198511.97 total 705.52 last 14.09) - open/close 14091 (time 1775198522.15 total 715.69 last 18.38) - open/close 14150 (time 1775198532.17 total 725.72 last 5.89) - open/close 14305 (time 1775198542.29 total 735.84 last 15.31) - open/close 14547 (time 1775198553.03 total 746.58 last 22.55) - open/close 14711 (time 1775198563.66 total 757.21 last 15.43) - open/close 14980 (time 1775198573.85 total 767.40 last 26.39) - open/close 15124 (time 1775198585.11 total 778.66 last 12.79) - open/close 15377 (time 1775198595.47 total 789.01 last 24.44) - open/close 15662 (time 1775198606.29 total 799.84 last 26.33) - open/close 15910 (time 1775198616.30 total 809.85 last 24.77) - open/close 16003 (time 1775198626.51 total 820.06 last 9.11) - open/close 16275 (time 1775198636.77 total 830.32 last 26.52) - open/close 16442 (time 1775198647.32 total 840.87 last 15.82) - open/close 16678 (time 1775198657.95 total 851.50 last 22.21) - open/close 16904 (time 1775198668.49 total 862.04 last 21.44) - open/close 17097 (time 1775198678.77 total 872.32 last 18.78) - open/close 17240 (time 1775198688.83 total 882.38 last 14.21) - open/close 17365 (time 1775198699.00 total 892.55 last 12.29) - open/close 17515 (time 1775198709.68 total 903.23 last 14.04) - open/close 17666 (time 1775198722.12 total 915.67 last 12.14) - open/close 17920 (time 1775198732.32 total 925.87 last 24.91) - open/close 18141 (time 1775198742.33 total 935.88 last 22.08) - open/close 18371 (time 1775198753.11 total 946.66 last 21.34) - open/close 18585 (time 1775198763.62 total 957.16 last 20.36) - open/close 18839 (time 1775198773.62 total 967.17 last 25.38) - open/close 19590 (time 1775198783.65 total 977.19 last 74.92) - open/close 20000 (time 1775198789.94 total 983.49 last 65.14) - open/close 20689 (time 1775198799.94 total 993.49 last 68.87) - open/close 21426 (time 1775198809.95 total 1003.50 last 73.68) - open/close 22111 (time 1775198819.97 total 1013.52 last 68.36) - open/close 22784 (time 1775198829.98 total 1023.53 last 67.23) - open/close 23430 (time 1775198839.99 total 1033.54 last 64.54) - open/close 24099 (time 1775198849.99 total 1043.54 last 66.89) - open/close 24755 (time 1775198859.99 total 1053.54 last 65.58) - open/close 25450 (time 1775198869.99 total 1063.54 last 69.49) - open/close 26193 (time 1775198880.01 total 1073.56 last 74.19) - open/close 26888 (time 1775198890.02 total 1083.57 last 69.41) - open/close 27633 (time 1775198900.03 total 1093.58 last 74.43) - open/close 28274 (time 1775198910.04 total 1103.59 last 64.05) - open/close 28923 (time 1775198920.05 total 1113.59 last 64.86) - open/close 29591 (time 1775198930.06 total 1123.61 last 66.69) - open/close 30000 (time 1775198935.36 total 1128.91 last 77.24) - open/close 30628 (time 1775198945.38 total 1138.93 last 62.62) - open/close 31244 (time 1775198955.38 total 1148.93 last 61.60) - open/close 31876 (time 1775198965.39 total 1158.94 last 63.15) - open/close 32503 (time 1775198975.41 total 1168.95 last 62.62) - open/close 33061 (time 1775198985.41 total 1178.96 last 55.78) - open/close 33839 (time 1775198995.42 total 1188.96 last 77.75) - open/close 34452 (time 1775199005.43 total 1198.98 last 61.23) - open/close 35198 (time 1775199015.44 total 1208.99 last 74.53) - open/close 35875 (time 1775199025.44 total 1218.99 last 67.66) - open/close 36464 (time 1775199035.45 total 1228.99 last 58.88) - open/close 37090 (time 1775199045.45 total 1239.00 last 62.57) - open/close 37680 (time 1775199055.46 total 1249.01 last 58.91) - open/close 38360 (time 1775199065.49 total 1259.04 last 67.84) - open/close 39031 (time 1775199075.49 total 1269.04 last 67.10) - open/close 39721 (time 1775199085.50 total 1279.05 last 68.92) - open/close 40000 (time 1775199089.13 total 1282.68 last 76.77) - open/close 40656 (time 1775199099.14 total 1292.69 last 65.54) - open/close 41410 (time 1775199109.16 total 1302.71 last 75.24) - open/close 42000 (time 1775199119.17 total 1312.72 last 58.97) - open/close 42649 (time 1775199129.18 total 1322.73 last 64.84) - open/close 43201 (time 1775199139.18 total 1332.73 last 55.20) - open/close 43727 (time 1775199149.19 total 1342.74 last 52.55) - open/close 44363 (time 1775199159.19 total 1352.74 last 63.56) - open/close 44973 (time 1775199169.19 total 1362.74 last 61.00) - open/close 45600 (time 1775199179.20 total 1372.75 last 62.68) - open/close 46266 (time 1775199189.20 total 1382.75 last 66.57) - open/close 46881 (time 1775199199.21 total 1392.76 last 61.44) - open/close 47557 (time 1775199209.22 total 1402.77 last 67.51) - open/close 48176 (time 1775199219.24 total 1412.79 last 61.82) - open/close 48876 (time 1775199229.26 total 1422.81 last 69.87) - open/close 49499 (time 1775199239.27 total 1432.81 last 62.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1435.05 seconds: 34.59 ops/second - unlinked 0 (time 1775199246 ; total 0 ; last 0) - unlinked 10000 (time 1775199398 ; total 152 ; last 152) - unlinked 20000 (time 1775199554 ; total 308 ; last 156) - unlinked 30000 (time 1775199711 ; total 465 ; last 157) - unlinked 40000 (time 1775199873 ; total 627 ; last 162) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 814 seconds: 60.972973 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2