== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:06:08 (1784048768) start mds service on oleg407-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 161 (time 1784048890.72 total 10.35 last 15.56) - open/close 367 (time 1784048900.74 total 20.37 last 20.56) - open/close 613 (time 1784048911.45 total 31.08 last 22.96) - open/close 858 (time 1784048922.02 total 41.65 last 23.19) - open/close 1153 (time 1784048932.02 total 51.65 last 29.48) - open/close 1393 (time 1784048942.03 total 61.66 last 23.99) - open/close 1486 (time 1784048952.47 total 72.10 last 8.91) - open/close 1738 (time 1784048962.48 total 82.11 last 25.17) - open/close 2007 (time 1784048973.12 total 92.75 last 25.29) - open/close 2199 (time 1784048983.83 total 103.46 last 17.93) - open/close 2491 (time 1784048993.85 total 113.48 last 29.14) - open/close 2745 (time 1784049003.85 total 123.48 last 25.40) - open/close 2966 (time 1784049013.87 total 133.50 last 22.06) - open/close 3206 (time 1784049023.89 total 143.51 last 23.96) - open/close 3374 (time 1784049034.07 total 153.70 last 16.50) - open/close 3628 (time 1784049044.07 total 163.70 last 25.40) - open/close 3848 (time 1784049054.07 total 173.70 last 21.99) - open/close 4089 (time 1784049064.66 total 184.28 last 22.78) - open/close 4313 (time 1784049074.70 total 194.33 last 22.30) - open/close 4560 (time 1784049084.84 total 204.47 last 24.36) - open/close 4825 (time 1784049095.83 total 215.46 last 24.12) - open/close 5085 (time 1784049106.46 total 226.09 last 24.45) - open/close 5333 (time 1784049117.46 total 237.09 last 22.55) - open/close 5588 (time 1784049128.23 total 247.86 last 23.68) - open/close 5846 (time 1784049139.12 total 258.75 last 23.70) - open/close 6156 (time 1784049149.97 total 269.60 last 28.56) - open/close 6399 (time 1784049159.97 total 279.60 last 24.29) - open/close 6617 (time 1784049169.99 total 289.62 last 21.77) - open/close 6769 (time 1784049180.00 total 299.63 last 15.19) - open/close 7013 (time 1784049190.09 total 309.72 last 24.18) - open/close 7139 (time 1784049200.49 total 320.12 last 12.12) - open/close 7295 (time 1784049210.80 total 330.43 last 15.14) - open/close 7517 (time 1784049222.27 total 341.90 last 19.35) - open/close 7773 (time 1784049232.91 total 352.54 last 24.05) - open/close 8052 (time 1784049243.29 total 362.92 last 26.89) - open/close 8208 (time 1784049253.30 total 372.93 last 15.58) - open/close 8400 (time 1784049263.57 total 383.20 last 18.68) - open/close 8633 (time 1784049274.46 total 394.09 last 21.41) - open/close 8765 (time 1784049284.46 total 404.09 last 13.19) - open/close 8953 (time 1784049294.47 total 414.10 last 18.78) - open/close 9128 (time 1784049305.12 total 424.75 last 16.44) - open/close 9304 (time 1784049316.19 total 435.82 last 15.89) - open/close 9462 (time 1784049327.07 total 446.69 last 14.54) - open/close 9621 (time 1784049338.01 total 457.64 last 14.52) - open/close 9771 (time 1784049348.56 total 468.19 last 14.22) - open/close 9846 (time 1784049358.70 total 478.33 last 7.40) - open/close 10000 (time 1784049368.40 total 488.03 last 15.88) - open/close 10135 (time 1784049378.72 total 498.35 last 13.08) - open/close 10323 (time 1784049389.02 total 508.65 last 18.26) - open/close 10395 (time 1784049399.65 total 519.28 last 6.77) - open/close 10543 (time 1784049410.68 total 530.30 last 13.42) - open/close 10657 (time 1784049420.98 total 540.61 last 11.06) - open/close 10736 (time 1784049431.14 total 550.77 last 7.78) - open/close 10922 (time 1784049441.75 total 561.37 last 17.53) - open/close 11155 (time 1784049452.77 total 572.40 last 21.14) - open/close 11383 (time 1784049462.89 total 582.52 last 22.52) - open/close 11457 (time 1784049473.82 total 593.45 last 6.77) - open/close 11641 (time 1784049483.85 total 603.48 last 18.35) - open/close 11865 (time 1784049493.86 total 613.49 last 22.38) - open/close 12101 (time 1784049503.86 total 623.49 last 23.59) - open/close 12344 (time 1784049513.87 total 633.50 last 24.28) - open/close 12547 (time 1784049525.16 total 644.79 last 17.98) - open/close 12757 (time 1784049535.80 total 655.43 last 19.73) - open/close 12898 (time 1784049545.81 total 665.44 last 14.09) - open/close 13123 (time 1784049555.82 total 675.45 last 22.48) - open/close 13399 (time 1784049565.83 total 685.46 last 27.55) - open/close 13634 (time 1784049578.39 total 698.02 last 18.72) - open/close 13810 (time 1784049588.40 total 708.03 last 17.59) - open/close 13929 (time 1784049598.41 total 718.04 last 11.89) - open/close 14070 (time 1784049608.42 total 728.04 last 14.09) - open/close 14232 (time 1784049618.85 total 738.48 last 15.52) - open/close 14359 (time 1784049629.01 total 748.64 last 12.50) - open/close 14538 (time 1784049639.73 total 759.36 last 16.70) - open/close 14825 (time 1784049650.27 total 769.90 last 27.23) - open/close 15102 (time 1784049660.27 total 779.90 last 27.69) - open/close 15344 (time 1784049670.29 total 789.92 last 24.15) - open/close 15540 (time 1784049680.31 total 799.94 last 19.57) - open/close 15741 (time 1784049690.68 total 810.30 last 19.39) - open/close 15891 (time 1784049703.03 total 822.66 last 12.14) - open/close 15975 (time 1784049714.08 total 833.71 last 7.61) - open/close 16218 (time 1784049724.09 total 843.72 last 24.26) - open/close 16406 (time 1784049734.28 total 853.91 last 18.45) - open/close 16637 (time 1784049744.29 total 863.92 last 23.07) - open/close 16747 (time 1784049754.89 total 874.52 last 10.38) - open/close 16811 (time 1784049765.92 total 885.55 last 5.80) - open/close 16953 (time 1784049775.94 total 895.57 last 14.17) - open/close 17176 (time 1784049785.94 total 905.57 last 22.30) - open/close 17295 (time 1784049795.94 total 915.57 last 11.90) - open/close 17422 (time 1784049805.96 total 925.59 last 12.68) - open/close 17638 (time 1784049816.40 total 936.03 last 20.68) - open/close 17846 (time 1784049829.05 total 948.68 last 16.45) - open/close 18013 (time 1784049839.06 total 958.69 last 16.68) - open/close 18158 (time 1784049849.11 total 968.74 last 14.42) - open/close 18316 (time 1784049859.12 total 978.75 last 15.80) - open/close 18884 (time 1784049869.12 total 988.75 last 56.77) - open/close 19323 (time 1784049879.15 total 998.78 last 43.76) - open/close 19687 (time 1784049889.15 total 1008.78 last 36.39) - open/close 20000 (time 1784049896.30 total 1015.93 last 43.79) - open/close 20479 (time 1784049906.31 total 1025.94 last 47.87) - open/close 21034 (time 1784049916.33 total 1035.96 last 55.39) - open/close 21621 (time 1784049926.34 total 1045.97 last 58.64) - open/close 22257 (time 1784049936.35 total 1055.98 last 63.55) - open/close 22940 (time 1784049946.36 total 1065.99 last 68.20) - open/close 23635 (time 1784049956.36 total 1075.99 last 69.47) - open/close 24260 (time 1784049966.39 total 1086.02 last 62.32) - open/close 24806 (time 1784049976.40 total 1096.03 last 54.56) - open/close 25412 (time 1784049986.41 total 1106.04 last 60.53) - open/close 26048 (time 1784049996.42 total 1116.05 last 63.54) - open/close 26686 (time 1784050006.42 total 1126.05 last 63.78) - open/close 27353 (time 1784050016.44 total 1136.07 last 66.58) - open/close 27968 (time 1784050026.45 total 1146.08 last 61.46) - open/close 28633 (time 1784050036.46 total 1156.09 last 66.42) - open/close 29222 (time 1784050046.48 total 1166.11 last 58.78) - open/close 29924 (time 1784050056.49 total 1176.12 last 70.10) - open/close 30533 (time 1784050066.50 total 1186.13 last 60.87) - open/close 30987 (time 1784050076.50 total 1196.13 last 45.39) - open/close 31382 (time 1784050086.51 total 1206.14 last 39.48) - open/close 31858 (time 1784050096.51 total 1216.14 last 47.59) - open/close 32520 (time 1784050106.53 total 1226.16 last 66.09) - open/close 33125 (time 1784050116.54 total 1236.17 last 60.45) - open/close 33811 (time 1784050126.56 total 1246.19 last 68.44) - open/close 34502 (time 1784050136.57 total 1256.20 last 69.04) - open/close 35227 (time 1784050146.58 total 1266.21 last 72.39) - open/close 35917 (time 1784050156.59 total 1276.22 last 68.97) - open/close 36567 (time 1784050166.61 total 1286.24 last 64.87) - open/close 37263 (time 1784050176.61 total 1296.24 last 69.56) - open/close 37943 (time 1784050186.63 total 1306.26 last 67.88) - open/close 38688 (time 1784050196.64 total 1316.27 last 74.46) - open/close 39366 (time 1784050206.65 total 1326.28 last 67.71) - open/close 40000 (time 1784050216.03 total 1335.66 last 67.61) - open/close 40637 (time 1784050226.03 total 1345.66 last 63.65) - open/close 41294 (time 1784050236.04 total 1355.67 last 65.64) - open/close 41955 (time 1784050246.05 total 1365.68 last 66.03) - open/close 42621 (time 1784050256.07 total 1375.70 last 66.49) - open/close 43288 (time 1784050266.07 total 1385.70 last 66.70) - open/close 43933 (time 1784050276.07 total 1395.70 last 64.50) - open/close 44533 (time 1784050286.08 total 1405.71 last 59.95) - open/close 45172 (time 1784050296.08 total 1415.71 last 63.86) - open/close 45864 (time 1784050306.09 total 1425.72 last 69.17) - open/close 46510 (time 1784050316.10 total 1435.73 last 64.53) - open/close 47149 (time 1784050326.11 total 1445.74 last 63.86) - open/close 47814 (time 1784050336.12 total 1455.75 last 66.41) - open/close 48520 (time 1784050346.14 total 1465.77 last 70.49) - open/close 49185 (time 1784050356.15 total 1475.78 last 66.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1483.21 seconds: 33.46 ops/second - unlinked 0 (time 1784050368 ; total 0 ; last 0) - unlinked 10000 (time 1784050526 ; total 158 ; last 158) - unlinked 20000 (time 1784050687 ; total 319 ; last 161) - unlinked 30000 (time 1784050843 ; total 475 ; last 156) - unlinked 40000 (time 1784050999 ; total 631 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 776 seconds: 63.957474 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2