== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:21:33 (1787869293) start mds service on oleg110-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg110-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg110-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg110-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg110-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 173 (time 1787869399.58 total 10.23 last 16.90) - open/close 458 (time 1787869410.19 total 20.85 last 26.84) - open/close 665 (time 1787869420.49 total 31.15 last 20.10) - open/close 857 (time 1787869430.51 total 41.17 last 19.17) - open/close 1103 (time 1787869440.51 total 51.17 last 24.60) - open/close 1391 (time 1787869451.55 total 62.21 last 26.09) - open/close 1664 (time 1787869461.56 total 72.21 last 27.28) - open/close 1909 (time 1787869472.10 total 82.76 last 23.23) - open/close 2127 (time 1787869482.41 total 93.07 last 21.16) - open/close 2360 (time 1787869492.70 total 103.35 last 22.65) - open/close 2583 (time 1787869502.94 total 113.60 last 21.77) - open/close 2882 (time 1787869512.94 total 123.60 last 29.88) - open/close 3140 (time 1787869523.48 total 134.13 last 24.50) - open/close 3432 (time 1787869534.45 total 145.11 last 26.60) - open/close 3701 (time 1787869544.46 total 155.11 last 26.89) - open/close 3971 (time 1787869554.93 total 165.59 last 25.77) - open/close 4143 (time 1787869565.53 total 176.19 last 16.22) - open/close 4471 (time 1787869576.24 total 186.90 last 30.64) - open/close 4751 (time 1787869587.17 total 197.83 last 25.62) - open/close 4980 (time 1787869597.18 total 207.84 last 22.88) - open/close 5288 (time 1787869607.69 total 218.35 last 29.30) - open/close 5592 (time 1787869618.15 total 228.81 last 29.07) - open/close 5822 (time 1787869628.16 total 238.82 last 22.98) - open/close 6091 (time 1787869638.17 total 248.82 last 26.88) - open/close 6267 (time 1787869648.41 total 259.07 last 17.17) - open/close 6438 (time 1787869659.29 total 269.95 last 15.72) - open/close 6693 (time 1787869669.31 total 279.97 last 25.45) - open/close 6939 (time 1787869679.92 total 290.58 last 23.19) - open/close 7192 (time 1787869690.20 total 300.86 last 24.61) - open/close 7401 (time 1787869700.38 total 311.04 last 20.53) - open/close 7604 (time 1787869711.39 total 322.05 last 18.43) - open/close 7735 (time 1787869722.06 total 332.72 last 12.28) - open/close 7999 (time 1787869732.07 total 342.73 last 26.37) - open/close 8245 (time 1787869742.08 total 352.74 last 24.58) - open/close 8529 (time 1787869752.99 total 363.65 last 26.04) - open/close 8772 (time 1787869763.94 total 374.60 last 22.19) - open/close 8966 (time 1787869773.94 total 384.60 last 19.38) - open/close 9168 (time 1787869783.95 total 394.61 last 20.19) - open/close 9401 (time 1787869794.98 total 405.64 last 21.13) - open/close 9632 (time 1787869805.12 total 415.78 last 22.78) - open/close 9902 (time 1787869815.12 total 425.78 last 26.99) - open/close 10000 (time 1787869819.05 total 429.70 last 24.96) - open/close 10250 (time 1787869830.07 total 440.72 last 22.69) - open/close 10439 (time 1787869840.16 total 450.82 last 18.73) - open/close 10714 (time 1787869850.16 total 460.82 last 27.50) - open/close 10895 (time 1787869860.18 total 470.83 last 18.07) - open/close 11125 (time 1787869870.19 total 480.85 last 22.97) - open/close 11366 (time 1787869881.25 total 491.91 last 21.79) - open/close 11541 (time 1787869892.58 total 503.24 last 15.45) - open/close 11805 (time 1787869902.81 total 513.47 last 25.79) - open/close 12039 (time 1787869912.82 total 523.48 last 23.39) - open/close 12309 (time 1787869923.38 total 534.04 last 25.56) - open/close 12507 (time 1787869933.59 total 544.25 last 19.39) - open/close 12645 (time 1787869943.64 total 554.30 last 13.73) - open/close 12847 (time 1787869953.84 total 564.49 last 19.81) - open/close 12998 (time 1787869964.71 total 575.37 last 13.89) - open/close 13224 (time 1787869974.72 total 585.38 last 22.57) - open/close 13466 (time 1787869985.31 total 595.96 last 22.87) - open/close 13700 (time 1787869995.32 total 605.97 last 23.38) - open/close 13887 (time 1787870005.98 total 616.64 last 17.53) - open/close 14181 (time 1787870016.00 total 626.66 last 29.34) - open/close 14430 (time 1787870026.59 total 637.25 last 23.51) - open/close 14669 (time 1787870036.60 total 647.26 last 23.87) - open/close 14917 (time 1787870046.80 total 657.46 last 24.32) - open/close 15188 (time 1787870056.81 total 667.47 last 27.07) - open/close 15337 (time 1787870067.68 total 678.34 last 13.71) - open/close 15621 (time 1787870077.70 total 688.36 last 28.34) - open/close 15861 (time 1787870087.72 total 698.38 last 23.96) - open/close 16067 (time 1787870099.45 total 710.11 last 17.56) - open/close 16216 (time 1787870109.82 total 720.48 last 14.36) - open/close 16379 (time 1787870120.55 total 731.21 last 15.20) - open/close 16567 (time 1787870131.45 total 742.11 last 17.25) - open/close 16817 (time 1787870141.45 total 752.11 last 24.99) - open/close 17060 (time 1787870151.99 total 762.65 last 23.06) - open/close 17301 (time 1787870162.91 total 773.57 last 22.07) - open/close 17610 (time 1787870172.91 total 783.57 last 30.90) - open/close 17781 (time 1787870183.97 total 794.62 last 15.47) - open/close 17896 (time 1787870194.64 total 805.30 last 10.77) - open/close 18062 (time 1787870205.15 total 815.81 last 15.79) - open/close 18195 (time 1787870215.83 total 826.49 last 12.45) - open/close 18487 (time 1787870225.84 total 836.49 last 29.20) - open/close 18712 (time 1787870235.85 total 846.51 last 22.46) - open/close 18993 (time 1787870246.04 total 856.70 last 27.59) - open/close 19211 (time 1787870257.50 total 868.16 last 19.02) - open/close 19487 (time 1787870267.51 total 878.17 last 27.58) - open/close 19625 (time 1787870277.51 total 888.17 last 13.80) - open/close 19870 (time 1787870288.55 total 899.21 last 22.19) - open/close 19958 (time 1787870299.03 total 909.69 last 8.40) - open/close 20000 (time 1787870304.32 total 914.98 last 7.94) - open/close 20130 (time 1787870317.16 total 927.81 last 10.13) - open/close 20387 (time 1787870327.16 total 937.82 last 25.69) - open/close 21024 (time 1787870337.17 total 947.83 last 63.63) - open/close 21799 (time 1787870347.18 total 957.83 last 77.46) - open/close 22507 (time 1787870357.18 total 967.84 last 70.77) - open/close 23118 (time 1787870367.18 total 977.84 last 61.06) - open/close 23925 (time 1787870377.19 total 987.85 last 80.63) - open/close 24642 (time 1787870387.21 total 997.87 last 71.56) - open/close 25430 (time 1787870397.22 total 1007.88 last 78.75) - open/close 26329 (time 1787870407.23 total 1017.88 last 89.83) - open/close 27012 (time 1787870417.23 total 1027.89 last 68.28) - open/close 27901 (time 1787870427.23 total 1037.89 last 88.88) - open/close 28789 (time 1787870437.24 total 1047.90 last 88.72) - open/close 29639 (time 1787870447.25 total 1057.90 last 84.95) - open/close 30000 (time 1787870450.67 total 1061.33 last 105.28) - open/close 31062 (time 1787870460.68 total 1071.33 last 106.19) - open/close 31902 (time 1787870470.68 total 1081.34 last 83.96) - open/close 32901 (time 1787870480.69 total 1091.34 last 99.85) - open/close 33648 (time 1787870490.69 total 1101.35 last 74.65) - open/close 34540 (time 1787870500.70 total 1111.35 last 89.17) - open/close 35470 (time 1787870510.70 total 1121.36 last 92.99) - open/close 36441 (time 1787870520.70 total 1131.36 last 97.05) - open/close 37369 (time 1787870530.70 total 1141.36 last 92.78) - open/close 38283 (time 1787870540.71 total 1151.36 last 91.38) - open/close 39303 (time 1787870550.71 total 1161.37 last 101.92) - open/close 40000 (time 1787870557.47 total 1168.12 last 103.21) - open/close 41064 (time 1787870567.47 total 1178.13 last 106.35) - open/close 42075 (time 1787870577.48 total 1188.14 last 101.04) - open/close 42872 (time 1787870587.48 total 1198.14 last 79.66) - open/close 43733 (time 1787870597.49 total 1208.15 last 86.04) - open/close 44382 (time 1787870607.50 total 1218.16 last 64.81) - open/close 44840 (time 1787870617.52 total 1228.18 last 45.73) - open/close 45363 (time 1787870627.55 total 1238.21 last 52.14) - open/close 46066 (time 1787870637.55 total 1248.21 last 70.30) - open/close 46881 (time 1787870647.56 total 1258.22 last 81.41) - open/close 47643 (time 1787870657.56 total 1268.22 last 76.16) - open/close 48452 (time 1787870667.57 total 1278.23 last 80.86) - open/close 49339 (time 1787870677.58 total 1288.24 last 88.61) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1292.38 seconds: 38.40 ops/second - unlinked 0 (time 1787870686 ; total 0 ; last 0) - unlinked 10000 (time 1787870795 ; total 109 ; last 109) - unlinked 20000 (time 1787870899 ; total 213 ; last 104) - unlinked 30000 (time 1787870959 ; total 273 ; last 60) - unlinked 40000 (time 1787871018 ; total 332 ; last 59) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 387 seconds: 128.242889 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server unloading modules via unload_modules_local on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local oleg110-server: modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2