== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:42:06 (1787823726) start mds service on oleg227-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg227-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 162 (time 1787823853.77 total 11.49 last 14.10) - open/close 375 (time 1787823863.77 total 21.50 last 21.29) - open/close 638 (time 1787823874.78 total 32.50 last 23.90) - open/close 901 (time 1787823885.56 total 43.29 last 24.39) - open/close 1123 (time 1787823896.00 total 53.72 last 21.28) - open/close 1383 (time 1787823906.67 total 64.40 last 24.35) - open/close 1638 (time 1787823916.68 total 74.41 last 25.48) - open/close 1851 (time 1787823926.69 total 84.41 last 21.28) - open/close 1999 (time 1787823937.04 total 94.77 last 14.30) - open/close 2073 (time 1787823947.94 total 105.67 last 6.79) - open/close 2172 (time 1787823958.48 total 116.20 last 9.39) - open/close 2323 (time 1787823968.49 total 126.21 last 15.08) - open/close 2413 (time 1787823978.62 total 136.34 last 8.89) - open/close 2558 (time 1787823989.42 total 147.14 last 13.43) - open/close 2744 (time 1787824000.47 total 158.20 last 16.83) - open/close 2845 (time 1787824011.40 total 169.13 last 9.24) - open/close 2971 (time 1787824022.49 total 180.21 last 11.37) - open/close 3109 (time 1787824032.52 total 190.25 last 13.75) - open/close 3295 (time 1787824042.64 total 200.37 last 18.37) - open/close 3487 (time 1787824052.65 total 210.38 last 19.19) - open/close 3682 (time 1787824063.31 total 221.03 last 18.30) - open/close 3871 (time 1787824073.54 total 231.26 last 18.48) - open/close 3982 (time 1787824084.18 total 241.91 last 10.42) - open/close 4221 (time 1787824094.21 total 251.93 last 23.84) - open/close 4432 (time 1787824104.79 total 262.51 last 19.94) - open/close 4624 (time 1787824116.10 total 273.82 last 16.98) - open/close 4805 (time 1787824127.18 total 284.90 last 16.34) - open/close 4974 (time 1787824137.18 total 294.91 last 16.89) - open/close 5105 (time 1787824149.26 total 306.99 last 10.84) - open/close 5246 (time 1787824159.81 total 317.54 last 13.36) - open/close 5483 (time 1787824170.31 total 328.04 last 22.58) - open/close 5755 (time 1787824181.23 total 338.96 last 24.91) - open/close 5869 (time 1787824191.30 total 349.02 last 11.32) - open/close 5958 (time 1787824201.54 total 359.27 last 8.69) - open/close 6078 (time 1787824211.73 total 369.45 last 11.78) - open/close 6247 (time 1787824223.74 total 381.47 last 14.06) - open/close 6370 (time 1787824233.75 total 391.48 last 12.29) - open/close 6455 (time 1787824244.03 total 401.75 last 8.27) - open/close 6514 (time 1787824254.09 total 411.81 last 5.86) - open/close 6635 (time 1787824264.46 total 422.19 last 11.66) - open/close 6885 (time 1787824274.50 total 432.22 last 24.92) - open/close 7097 (time 1787824285.51 total 443.23 last 19.26) - open/close 7192 (time 1787824296.01 total 453.74 last 9.04) - open/close 7312 (time 1787824306.69 total 464.41 last 11.24) - open/close 7395 (time 1787824317.13 total 474.86 last 7.95) - open/close 7575 (time 1787824327.76 total 485.48 last 16.95) - open/close 7708 (time 1787824338.00 total 495.72 last 12.98) - open/close 7765 (time 1787824348.88 total 506.60 last 5.24) - open/close 7953 (time 1787824359.43 total 517.15 last 17.82) - open/close 8164 (time 1787824369.54 total 527.26 last 20.87) - open/close 8318 (time 1787824380.69 total 538.42 last 13.81) - open/close 8567 (time 1787824390.98 total 548.70 last 24.21) - open/close 8783 (time 1787824401.17 total 558.89 last 21.20) - open/close 8990 (time 1787824411.97 total 569.70 last 19.15) - open/close 9192 (time 1787824422.11 total 579.83 last 19.93) - open/close 9330 (time 1787824436.05 total 593.77 last 9.90) - open/close 9550 (time 1787824446.66 total 604.39 last 20.72) - open/close 9777 (time 1787824456.66 total 614.39 last 22.69) - open/close 9954 (time 1787824468.36 total 626.09 last 15.13) - open/close 10181 (time 1787824479.13 total 636.85 last 21.09) - open/close 10335 (time 1787824490.05 total 647.77 last 14.10) - open/close 10490 (time 1787824500.05 total 657.78 last 15.49) - open/close 10655 (time 1787824510.24 total 667.97 last 16.20) - open/close 10881 (time 1787824520.24 total 677.97 last 22.60) - open/close 11009 (time 1787824530.56 total 688.29 last 12.40) - open/close 11218 (time 1787824540.57 total 698.29 last 20.90) - open/close 11433 (time 1787824550.58 total 708.30 last 21.47) - open/close 11722 (time 1787824561.12 total 718.84 last 27.43) - open/close 11928 (time 1787824571.14 total 728.86 last 20.56) - open/close 12108 (time 1787824581.58 total 739.30 last 17.24) - open/close 12374 (time 1787824591.59 total 749.31 last 26.57) - open/close 12555 (time 1787824602.32 total 760.05 last 16.86) - open/close 12803 (time 1787824612.34 total 770.07 last 24.76) - open/close 13042 (time 1787824622.37 total 780.09 last 23.84) - open/close 13208 (time 1787824632.50 total 790.23 last 16.37) - open/close 13416 (time 1787824642.51 total 800.23 last 20.80) - open/close 13655 (time 1787824653.54 total 811.27 last 21.65) - open/close 13888 (time 1787824663.62 total 821.35 last 23.12) - open/close 14029 (time 1787824674.62 total 832.35 last 12.82) - open/close 14206 (time 1787824684.94 total 842.66 last 17.16) - open/close 14444 (time 1787824695.31 total 853.03 last 22.96) - open/close 14706 (time 1787824705.37 total 863.09 last 26.03) - open/close 14903 (time 1787824715.38 total 873.10 last 19.68) - open/close 15137 (time 1787824725.62 total 883.35 last 22.84) - open/close 15199 (time 1787824738.37 total 896.09 last 4.87) - open/close 15443 (time 1787824748.38 total 906.10 last 24.38) - open/close 15667 (time 1787824758.78 total 916.50 last 21.53) - open/close 15908 (time 1787824769.41 total 927.13 last 22.67) - open/close 16153 (time 1787824782.22 total 939.94 last 19.13) - open/close 16341 (time 1787824792.24 total 949.97 last 18.75) - open/close 16620 (time 1787824802.25 total 959.97 last 27.89) - open/close 16808 (time 1787824812.26 total 969.99 last 18.77) - open/close 16945 (time 1787824822.74 total 980.46 last 13.08) - open/close 17038 (time 1787824833.29 total 991.01 last 8.81) - open/close 17180 (time 1787824843.31 total 1001.03 last 14.18) - open/close 17257 (time 1787824853.89 total 1011.62 last 7.27) - open/close 17447 (time 1787824864.45 total 1022.17 last 18.00) - open/close 18069 (time 1787824874.45 total 1032.17 last 62.18) - open/close 18928 (time 1787824884.46 total 1042.19 last 85.80) - open/close 19681 (time 1787824894.47 total 1052.19 last 75.27) - open/close 20000 (time 1787824898.94 total 1056.66 last 71.31) - open/close 20620 (time 1787824908.95 total 1066.68 last 61.91) - open/close 21186 (time 1787824918.96 total 1076.69 last 56.56) - open/close 22046 (time 1787824928.97 total 1086.69 last 85.92) - open/close 22790 (time 1787824938.97 total 1096.70 last 74.39) - open/close 23593 (time 1787824948.98 total 1106.70 last 80.24) - open/close 24305 (time 1787824958.98 total 1116.70 last 71.19) - open/close 24917 (time 1787824968.98 total 1126.71 last 61.18) - open/close 25595 (time 1787824978.99 total 1136.71 last 67.76) - open/close 26313 (time 1787824988.99 total 1146.72 last 71.77) - open/close 26999 (time 1787824999.00 total 1156.73 last 68.53) - open/close 27586 (time 1787825009.05 total 1166.78 last 58.40) - open/close 28260 (time 1787825019.07 total 1176.79 last 67.33) - open/close 28856 (time 1787825029.07 total 1186.80 last 59.56) - open/close 29604 (time 1787825039.07 total 1196.80 last 74.78) - open/close 30000 (time 1787825044.47 total 1202.19 last 73.40) - open/close 30834 (time 1787825054.48 total 1212.20 last 83.33) - open/close 31599 (time 1787825064.48 total 1222.20 last 76.50) - open/close 32334 (time 1787825074.49 total 1232.21 last 73.46) - open/close 33123 (time 1787825084.49 total 1242.21 last 78.88) - open/close 33945 (time 1787825094.49 total 1252.22 last 82.16) - open/close 34718 (time 1787825104.49 total 1262.22 last 77.28) - open/close 35490 (time 1787825114.50 total 1272.22 last 77.19) - open/close 36399 (time 1787825124.51 total 1282.23 last 90.79) - open/close 37298 (time 1787825134.52 total 1292.24 last 89.84) - open/close 38272 (time 1787825144.52 total 1302.25 last 97.33) - open/close 39120 (time 1787825154.53 total 1312.25 last 84.75) - open/close 39881 (time 1787825164.54 total 1322.26 last 76.01) - open/close 40621 (time 1787825174.54 total 1332.27 last 73.98) - open/close 41345 (time 1787825184.55 total 1342.28 last 72.34) - open/close 42110 (time 1787825194.56 total 1352.29 last 76.43) - open/close 42845 (time 1787825204.57 total 1362.30 last 73.43) - open/close 43636 (time 1787825214.58 total 1372.30 last 79.04) - open/close 44435 (time 1787825224.59 total 1382.31 last 79.84) - open/close 45213 (time 1787825234.59 total 1392.31 last 77.77) - open/close 45952 (time 1787825244.59 total 1402.32 last 73.88) - open/close 46786 (time 1787825254.60 total 1412.32 last 83.35) - open/close 47580 (time 1787825264.61 total 1422.33 last 79.33) - open/close 48369 (time 1787825274.62 total 1432.34 last 78.83) - open/close 49199 (time 1787825284.63 total 1442.36 last 82.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1448.61 seconds: 34.26 ops/second - unlinked 0 (time 1787825295 ; total 0 ; last 0) - unlinked 10000 (time 1787825444 ; total 149 ; last 149) - unlinked 20000 (time 1787825568 ; total 273 ; last 124) - unlinked 30000 (time 1787825699 ; total 404 ; last 131) - unlinked 40000 (time 1787825822 ; total 527 ; last 123) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 653 seconds: 76.004593 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: modules unloaded. pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2