== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:15:53 (1787390153) start mds service on oleg414-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg414-server: detected 4 online CPUs by sysfs oleg414-server: MODOPTS_LIBCFS= oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg414-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg414-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg414-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 217 (time 1787390261.83 total 10.36 last 20.95) - open/close 352 (time 1787390272.25 total 20.78 last 12.95) - open/close 445 (time 1787390282.43 total 30.95 last 9.14) - open/close 628 (time 1787390292.93 total 41.45 last 17.44) - open/close 798 (time 1787390303.61 total 52.14 last 15.91) - open/close 925 (time 1787390313.67 total 62.20 last 12.62) - open/close 1149 (time 1787390323.71 total 72.23 last 22.33) - open/close 1402 (time 1787390333.73 total 82.25 last 25.25) - open/close 1575 (time 1787390344.62 total 93.14 last 15.88) - open/close 1779 (time 1787390355.09 total 103.61 last 19.48) - open/close 1982 (time 1787390365.39 total 113.91 last 19.72) - open/close 2190 (time 1787390375.40 total 123.92 last 20.77) - open/close 2419 (time 1787390385.68 total 134.20 last 22.28) - open/close 2693 (time 1787390395.70 total 144.22 last 27.35) - open/close 2811 (time 1787390406.52 total 155.04 last 10.91) - open/close 3023 (time 1787390417.09 total 165.61 last 20.05) - open/close 3153 (time 1787390427.71 total 176.23 last 12.24) - open/close 3436 (time 1787390437.88 total 186.40 last 27.82) - open/close 3674 (time 1787390448.24 total 196.76 last 22.98) - open/close 3947 (time 1787390459.34 total 207.86 last 24.59) - open/close 4139 (time 1787390469.35 total 217.87 last 19.18) - open/close 4396 (time 1787390479.36 total 227.89 last 25.66) - open/close 4501 (time 1787390490.29 total 238.81 last 9.61) - open/close 4579 (time 1787390502.64 total 251.16 last 6.31) - open/close 4792 (time 1787390512.97 total 261.50 last 20.61) - open/close 4996 (time 1787390522.98 total 271.50 last 20.39) - open/close 5213 (time 1787390533.24 total 281.76 last 21.16) - open/close 5479 (time 1787390543.94 total 292.47 last 24.84) - open/close 5616 (time 1787390553.95 total 302.47 last 13.69) - open/close 5777 (time 1787390564.28 total 312.81 last 15.58) - open/close 5972 (time 1787390574.31 total 322.83 last 19.46) - open/close 6180 (time 1787390584.32 total 332.84 last 20.77) - open/close 6345 (time 1787390594.38 total 342.90 last 16.41) - open/close 6567 (time 1787390604.85 total 353.37 last 21.20) - open/close 6732 (time 1787390619.77 total 368.29 last 11.06) - open/close 6904 (time 1787390630.51 total 379.04 last 16.00) - open/close 7059 (time 1787390640.84 total 389.36 last 15.02) - open/close 7351 (time 1787390651.62 total 400.14 last 27.08) - open/close 7649 (time 1787390661.64 total 410.17 last 29.73) - open/close 7771 (time 1787390671.93 total 420.45 last 11.86) - open/close 7996 (time 1787390681.94 total 430.46 last 22.49) - open/close 8212 (time 1787390692.54 total 441.06 last 20.37) - open/close 8473 (time 1787390703.54 total 452.06 last 23.73) - open/close 8667 (time 1787390714.01 total 462.53 last 18.53) - open/close 8851 (time 1787390724.85 total 473.38 last 16.97) - open/close 9033 (time 1787390735.76 total 484.29 last 16.68) - open/close 9273 (time 1787390746.04 total 494.57 last 23.35) - open/close 9582 (time 1787390756.07 total 504.59 last 30.83) - open/close 9782 (time 1787390766.46 total 514.99 last 19.24) - open/close 9927 (time 1787390776.95 total 525.47 last 13.83) - open/close 10000 (time 1787390779.50 total 528.02 last 28.62) - open/close 10185 (time 1787390791.99 total 540.52 last 14.80) - open/close 10437 (time 1787390802.30 total 550.82 last 24.46) - open/close 10707 (time 1787390812.40 total 560.92 last 26.73) - open/close 10975 (time 1787390822.40 total 570.92 last 26.80) - open/close 11211 (time 1787390833.07 total 581.60 last 22.11) - open/close 11479 (time 1787390844.02 total 592.54 last 24.49) - open/close 11771 (time 1787390854.46 total 602.98 last 27.97) - open/close 11884 (time 1787390865.54 total 614.06 last 10.19) - open/close 12138 (time 1787390876.50 total 625.03 last 23.17) - open/close 12227 (time 1787390886.96 total 635.48 last 8.51) - open/close 12295 (time 1787390897.27 total 645.79 last 6.60) - open/close 12497 (time 1787390907.53 total 656.05 last 19.69) - open/close 12660 (time 1787390917.95 total 666.47 last 15.64) - open/close 12806 (time 1787390928.97 total 677.49 last 13.25) - open/close 12929 (time 1787390939.56 total 688.08 last 11.61) - open/close 13033 (time 1787390949.56 total 698.09 last 10.40) - open/close 13241 (time 1787390959.66 total 708.19 last 20.59) - open/close 13510 (time 1787390970.69 total 719.21 last 24.41) - open/close 13767 (time 1787390981.69 total 730.22 last 23.35) - open/close 13940 (time 1787390991.72 total 740.24 last 17.26) - open/close 14011 (time 1787391001.74 total 750.27 last 7.08) - open/close 14074 (time 1787391011.75 total 760.28 last 6.29) - open/close 14169 (time 1787391022.47 total 770.99 last 8.87) - open/close 14406 (time 1787391033.34 total 781.86 last 21.80) - open/close 14558 (time 1787391043.63 total 792.15 last 14.77) - open/close 14747 (time 1787391054.49 total 803.01 last 17.41) - open/close 15006 (time 1787391064.50 total 813.02 last 25.87) - open/close 15205 (time 1787391074.55 total 823.07 last 19.79) - open/close 15355 (time 1787391085.20 total 833.72 last 14.09) - open/close 15596 (time 1787391095.21 total 843.74 last 24.06) - open/close 15775 (time 1787391105.23 total 853.75 last 17.87) - open/close 15969 (time 1787391115.24 total 863.76 last 19.38) - open/close 16207 (time 1787391125.25 total 873.77 last 23.77) - open/close 16399 (time 1787391135.26 total 883.78 last 19.19) - open/close 16664 (time 1787391145.85 total 894.38 last 25.01) - open/close 16914 (time 1787391156.53 total 905.06 last 23.41) - open/close 17173 (time 1787391166.55 total 915.07 last 25.87) - open/close 17435 (time 1787391177.33 total 925.86 last 24.29) - open/close 17632 (time 1787391187.34 total 935.86 last 19.70) - open/close 17750 (time 1787391197.35 total 945.87 last 11.78) - open/close 17993 (time 1787391207.60 total 956.12 last 23.71) - open/close 18212 (time 1787391217.97 total 966.49 last 21.11) - open/close 18428 (time 1787391227.98 total 976.50 last 21.59) - open/close 18669 (time 1787391238.06 total 986.58 last 23.90) - open/close 18952 (time 1787391249.08 total 997.61 last 25.68) - open/close 19547 (time 1787391259.09 total 1007.61 last 59.47) - open/close 20000 (time 1787391265.39 total 1013.91 last 71.87) - open/close 20746 (time 1787391275.40 total 1023.92 last 74.57) - open/close 21543 (time 1787391285.41 total 1033.93 last 79.61) - open/close 22302 (time 1787391295.42 total 1043.94 last 75.83) - open/close 23106 (time 1787391305.42 total 1053.94 last 80.38) - open/close 23793 (time 1787391315.42 total 1063.94 last 68.69) - open/close 24508 (time 1787391325.44 total 1073.96 last 71.36) - open/close 25225 (time 1787391335.46 total 1083.98 last 71.58) - open/close 25929 (time 1787391345.46 total 1093.98 last 70.39) - open/close 26637 (time 1787391355.47 total 1104.00 last 70.68) - open/close 27375 (time 1787391365.49 total 1114.01 last 73.70) - open/close 28203 (time 1787391375.49 total 1124.02 last 82.75) - open/close 28974 (time 1787391385.49 total 1134.02 last 77.09) - open/close 29767 (time 1787391395.50 total 1144.02 last 79.28) - open/close 30000 (time 1787391398.42 total 1146.95 last 79.66) - open/close 30767 (time 1787391408.43 total 1156.96 last 76.62) - open/close 31470 (time 1787391418.44 total 1166.96 last 70.26) - open/close 32144 (time 1787391428.45 total 1176.97 last 67.32) - open/close 32830 (time 1787391438.47 total 1186.99 last 68.47) - open/close 33571 (time 1787391448.48 total 1197.00 last 74.05) - open/close 34252 (time 1787391458.48 total 1207.00 last 68.09) - open/close 34848 (time 1787391468.49 total 1217.01 last 59.55) - open/close 35510 (time 1787391478.49 total 1227.01 last 66.17) - open/close 36128 (time 1787391488.50 total 1237.02 last 61.76) - open/close 36664 (time 1787391498.51 total 1247.03 last 53.56) - open/close 37237 (time 1787391508.52 total 1257.04 last 57.23) - open/close 37822 (time 1787391518.52 total 1267.04 last 58.48) - open/close 38332 (time 1787391528.54 total 1277.06 last 50.93) - open/close 39035 (time 1787391538.54 total 1287.06 last 70.30) - open/close 39619 (time 1787391548.54 total 1297.06 last 58.39) - open/close 40000 (time 1787391554.49 total 1303.02 last 63.95) - open/close 40666 (time 1787391564.50 total 1313.02 last 66.57) - open/close 41443 (time 1787391574.51 total 1323.03 last 77.62) - open/close 42205 (time 1787391584.53 total 1333.05 last 76.07) - open/close 42958 (time 1787391594.53 total 1343.06 last 75.24) - open/close 43655 (time 1787391604.54 total 1353.06 last 69.66) - open/close 44293 (time 1787391614.55 total 1363.07 last 63.75) - open/close 45046 (time 1787391624.55 total 1373.07 last 75.27) - open/close 45688 (time 1787391634.58 total 1383.10 last 64.05) - open/close 46311 (time 1787391644.59 total 1393.11 last 62.23) - open/close 46971 (time 1787391654.59 total 1403.11 last 65.98) - open/close 47631 (time 1787391664.59 total 1413.12 last 65.98) - open/close 48376 (time 1787391674.60 total 1423.12 last 74.47) - open/close 49009 (time 1787391684.61 total 1433.13 last 63.23) - open/close 49624 (time 1787391694.62 total 1443.14 last 61.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1443.21 seconds: 34.39 ops/second - unlinked 0 (time 1787391699 ; total 0 ; last 0) - unlinked 10000 (time 1787391846 ; total 147 ; last 147) - unlinked 20000 (time 1787392006 ; total 307 ; last 160) - unlinked 30000 (time 1787392157 ; total 458 ; last 151) - unlinked 40000 (time 1787392299 ; total 600 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 734 seconds: 67.617165 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local oleg414-server: modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2