== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:11:04 (1787397064) start mds service on oleg142-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg142-server: detected 4 online CPUs by sysfs oleg142-server: MODOPTS_LIBCFS= oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg142-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg142-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg142-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg142-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1787397190.74 total 10.98 last 18.13) - open/close 428 (time 1787397201.81 total 22.04 last 20.69) - open/close 647 (time 1787397214.54 total 34.78 last 17.20) - open/close 814 (time 1787397225.12 total 45.36 last 15.78) - open/close 1018 (time 1787397235.91 total 56.15 last 18.91) - open/close 1071 (time 1787397246.34 total 66.58 last 5.08) - open/close 1357 (time 1787397257.03 total 77.27 last 26.75) - open/close 1532 (time 1787397267.04 total 87.28 last 17.49) - open/close 1697 (time 1787397277.32 total 97.56 last 16.04) - open/close 1914 (time 1787397287.61 total 107.85 last 21.09) - open/close 2064 (time 1787397297.80 total 118.03 last 14.73) - open/close 2273 (time 1787397308.69 total 128.93 last 19.18) - open/close 2304 (time 1787397320.84 total 141.07 last 2.55) - open/close 2509 (time 1787397330.84 total 151.08 last 20.49) - open/close 2655 (time 1787397340.84 total 161.08 last 14.59) - open/close 2775 (time 1787397351.50 total 171.74 last 11.26) - open/close 2954 (time 1787397361.54 total 181.78 last 17.82) - open/close 3149 (time 1787397372.44 total 192.68 last 17.90) - open/close 3406 (time 1787397385.41 total 205.64 last 19.82) - open/close 3522 (time 1787397395.41 total 215.65 last 11.59) - open/close 3612 (time 1787397405.57 total 225.81 last 8.86) - open/close 3773 (time 1787397416.47 total 236.71 last 14.77) - open/close 4012 (time 1787397426.47 total 246.71 last 23.89) - open/close 4231 (time 1787397436.49 total 256.72 last 21.87) - open/close 4455 (time 1787397446.62 total 266.85 last 22.11) - open/close 4689 (time 1787397456.73 total 276.97 last 23.14) - open/close 4926 (time 1787397466.74 total 286.98 last 23.67) - open/close 5150 (time 1787397477.54 total 297.78 last 20.74) - open/close 5329 (time 1787397487.55 total 307.79 last 17.88) - open/close 5407 (time 1787397497.74 total 317.98 last 7.66) - open/close 5607 (time 1787397508.36 total 328.60 last 18.83) - open/close 5880 (time 1787397521.43 total 341.67 last 20.88) - open/close 6149 (time 1787397531.43 total 351.67 last 26.89) - open/close 6241 (time 1787397541.66 total 361.89 last 9.00) - open/close 6453 (time 1787397552.01 total 372.25 last 20.47) - open/close 6536 (time 1787397562.11 total 382.35 last 8.22) - open/close 6659 (time 1787397572.12 total 392.35 last 12.30) - open/close 6815 (time 1787397582.73 total 402.96 last 14.70) - open/close 7029 (time 1787397592.74 total 412.98 last 21.37) - open/close 7174 (time 1787397603.08 total 423.32 last 14.02) - open/close 7419 (time 1787397613.87 total 434.11 last 22.71) - open/close 7480 (time 1787397624.83 total 445.07 last 5.56) - open/close 7654 (time 1787397635.30 total 455.54 last 16.62) - open/close 7949 (time 1787397645.31 total 465.55 last 29.48) - open/close 8185 (time 1787397655.31 total 475.55 last 23.59) - open/close 8465 (time 1787397668.35 total 488.59 last 21.48) - open/close 8707 (time 1787397678.37 total 498.61 last 24.14) - open/close 8839 (time 1787397689.35 total 509.59 last 12.03) - open/close 9089 (time 1787397699.65 total 519.89 last 24.26) - open/close 9204 (time 1787397711.51 total 531.75 last 9.70) - open/close 9312 (time 1787397722.17 total 542.40 last 10.13) - open/close 9481 (time 1787397732.17 total 552.41 last 16.89) - open/close 9680 (time 1787397742.20 total 562.44 last 19.84) - open/close 9829 (time 1787397752.28 total 572.52 last 14.78) - open/close 10000 (time 1787397759.43 total 579.66 last 23.95) - open/close 10217 (time 1787397772.03 total 592.27 last 17.21) - open/close 10297 (time 1787397782.04 total 602.28 last 8.00) - open/close 10555 (time 1787397792.05 total 612.29 last 25.77) - open/close 10829 (time 1787397802.06 total 622.30 last 27.37) - open/close 11047 (time 1787397812.07 total 632.31 last 21.78) - open/close 11221 (time 1787397823.06 total 643.29 last 15.84) - open/close 11475 (time 1787397833.64 total 653.88 last 24.00) - open/close 11766 (time 1787397844.29 total 664.53 last 27.31) - open/close 12029 (time 1787397854.87 total 675.11 last 24.86) - open/close 12270 (time 1787397864.88 total 685.11 last 24.09) - open/close 12478 (time 1787397874.88 total 695.12 last 20.79) - open/close 12665 (time 1787397887.69 total 707.93 last 14.59) - open/close 12799 (time 1787397897.70 total 717.94 last 13.40) - open/close 13090 (time 1787397907.72 total 727.96 last 29.02) - open/close 13289 (time 1787397918.60 total 738.84 last 18.30) - open/close 13469 (time 1787397928.61 total 748.84 last 17.99) - open/close 13664 (time 1787397938.67 total 758.90 last 19.38) - open/close 13889 (time 1787397950.60 total 770.84 last 18.85) - open/close 14117 (time 1787397960.61 total 780.85 last 22.79) - open/close 14324 (time 1787397970.61 total 790.85 last 20.70) - open/close 14568 (time 1787397980.61 total 800.85 last 24.39) - open/close 14734 (time 1787397991.04 total 811.28 last 15.92) - open/close 14981 (time 1787398001.86 total 822.10 last 22.83) - open/close 15238 (time 1787398012.74 total 832.97 last 23.64) - open/close 15461 (time 1787398022.86 total 843.10 last 22.03) - open/close 15688 (time 1787398033.68 total 853.91 last 20.99) - open/close 15873 (time 1787398044.29 total 864.53 last 17.42) - open/close 16143 (time 1787398054.30 total 874.54 last 26.98) - open/close 16417 (time 1787398065.03 total 885.27 last 25.54) - open/close 16666 (time 1787398075.04 total 895.28 last 24.87) - open/close 17001 (time 1787398085.46 total 905.70 last 32.15) - open/close 17253 (time 1787398095.80 total 916.04 last 24.38) - open/close 17543 (time 1787398106.67 total 926.91 last 26.68) - open/close 17824 (time 1787398117.50 total 937.74 last 25.93) - open/close 18097 (time 1787398127.51 total 947.75 last 27.27) - open/close 18325 (time 1787398137.52 total 957.76 last 22.78) - open/close 18544 (time 1787398147.81 total 968.05 last 21.29) - open/close 18758 (time 1787398157.82 total 978.06 last 21.38) - open/close 19014 (time 1787398168.70 total 988.94 last 23.52) - open/close 19428 (time 1787398178.70 total 998.94 last 41.39) - open/close 20000 (time 1787398185.37 total 1005.60 last 85.85) - open/close 20718 (time 1787398195.37 total 1015.61 last 71.79) - open/close 21486 (time 1787398205.38 total 1025.62 last 76.72) - open/close 22256 (time 1787398215.39 total 1035.63 last 76.91) - open/close 23022 (time 1787398225.40 total 1045.63 last 76.55) - open/close 23740 (time 1787398235.40 total 1055.64 last 71.74) - open/close 24536 (time 1787398245.42 total 1065.66 last 79.45) - open/close 25269 (time 1787398255.43 total 1075.67 last 73.24) - open/close 26002 (time 1787398265.44 total 1085.68 last 73.25) - open/close 26705 (time 1787398275.44 total 1095.68 last 70.26) - open/close 27431 (time 1787398285.45 total 1105.69 last 72.52) - open/close 28178 (time 1787398295.46 total 1115.70 last 74.65) - open/close 29054 (time 1787398305.47 total 1125.71 last 87.53) - open/close 29785 (time 1787398315.48 total 1135.71 last 73.05) - open/close 30000 (time 1787398317.94 total 1138.18 last 87.31) - open/close 30758 (time 1787398327.95 total 1148.19 last 75.73) - open/close 31463 (time 1787398337.95 total 1158.19 last 70.50) - open/close 32176 (time 1787398347.95 total 1168.19 last 71.25) - open/close 32887 (time 1787398357.97 total 1178.21 last 70.99) - open/close 33623 (time 1787398367.99 total 1188.23 last 73.47) - open/close 34212 (time 1787398378.00 total 1198.24 last 58.80) - open/close 34927 (time 1787398388.01 total 1208.25 last 71.48) - open/close 35712 (time 1787398398.01 total 1218.25 last 78.45) - open/close 36358 (time 1787398408.01 total 1228.25 last 64.59) - open/close 37021 (time 1787398418.02 total 1238.26 last 66.26) - open/close 37638 (time 1787398428.02 total 1248.26 last 61.68) - open/close 38355 (time 1787398438.04 total 1258.27 last 71.61) - open/close 39087 (time 1787398448.04 total 1268.27 last 73.20) - open/close 39807 (time 1787398458.04 total 1278.28 last 71.98) - open/close 40000 (time 1787398460.87 total 1281.11 last 68.22) - open/close 40726 (time 1787398470.87 total 1291.11 last 72.59) - open/close 41524 (time 1787398480.88 total 1301.12 last 79.72) - open/close 42160 (time 1787398490.90 total 1311.14 last 63.49) - open/close 42862 (time 1787398500.91 total 1321.15 last 70.11) - open/close 43494 (time 1787398510.91 total 1331.15 last 63.20) - open/close 44102 (time 1787398520.96 total 1341.20 last 60.53) - open/close 44729 (time 1787398530.97 total 1351.21 last 62.60) - open/close 45431 (time 1787398540.98 total 1361.22 last 70.15) - open/close 46139 (time 1787398550.98 total 1371.22 last 70.79) - open/close 46836 (time 1787398560.99 total 1381.23 last 69.62) - open/close 47535 (time 1787398571.02 total 1391.26 last 69.71) - open/close 48334 (time 1787398581.03 total 1401.27 last 79.82) - open/close 49001 (time 1787398591.04 total 1411.27 last 66.66) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1420.15 seconds: 34.95 ops/second - unlinked 0 (time 1787398605 ; total 0 ; last 0) - unlinked 10000 (time 1787398742 ; total 137 ; last 137) - unlinked 20000 (time 1787398891 ; total 286 ; last 149) - unlinked 30000 (time 1787399034 ; total 429 ; last 143) - unlinked 40000 (time 1787399181 ; total 576 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 714 seconds: 69.511208 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg142-server unloading modules via unload_modules_local on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local oleg142-server: modules unloaded. pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2