== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:11:58 (1787397118) start mds service on oleg104-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg104-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg104-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 276 (time 1787397233.29 total 10.71 last 25.78) - open/close 542 (time 1787397244.23 total 21.65 last 24.31) - open/close 835 (time 1787397254.24 total 31.66 last 29.29) - open/close 1043 (time 1787397265.16 total 42.57 last 19.05) - open/close 1127 (time 1787397275.28 total 52.69 last 8.30) - open/close 1320 (time 1787397285.84 total 63.26 last 18.27) - open/close 1399 (time 1787397296.71 total 74.13 last 7.26) - open/close 1579 (time 1787397306.83 total 84.24 last 17.80) - open/close 1719 (time 1787397318.93 total 96.35 last 11.57) - open/close 1959 (time 1787397329.25 total 106.66 last 23.26) - open/close 2209 (time 1787397339.97 total 117.39 last 23.31) - open/close 2470 (time 1787397350.29 total 127.70 last 25.31) - open/close 2723 (time 1787397360.30 total 137.71 last 25.27) - open/close 2960 (time 1787397370.32 total 147.74 last 23.64) - open/close 3216 (time 1787397380.50 total 157.92 last 25.16) - open/close 3384 (time 1787397390.52 total 167.94 last 16.76) - open/close 3455 (time 1787397401.98 total 179.39 last 6.20) - open/close 3658 (time 1787397412.84 total 190.25 last 18.69) - open/close 3820 (time 1787397422.85 total 200.26 last 16.18) - open/close 4081 (time 1787397433.89 total 211.30 last 23.64) - open/close 4385 (time 1787397444.68 total 222.10 last 28.16) - open/close 4701 (time 1787397455.44 total 232.85 last 29.39) - open/close 4932 (time 1787397465.93 total 243.35 last 22.01) - open/close 5144 (time 1787397475.93 total 253.35 last 21.20) - open/close 5428 (time 1787397486.61 total 264.03 last 26.60) - open/close 5675 (time 1787397496.63 total 274.05 last 24.65) - open/close 5942 (time 1787397506.64 total 284.06 last 26.68) - open/close 6249 (time 1787397516.64 total 294.06 last 30.70) - open/close 6448 (time 1787397526.74 total 304.16 last 19.70) - open/close 6580 (time 1787397536.74 total 314.16 last 13.20) - open/close 6803 (time 1787397547.35 total 324.77 last 21.02) - open/close 7078 (time 1787397557.90 total 335.32 last 26.05) - open/close 7194 (time 1787397567.97 total 345.38 last 11.53) - open/close 7409 (time 1787397577.97 total 355.39 last 21.49) - open/close 7675 (time 1787397587.97 total 365.39 last 26.59) - open/close 7908 (time 1787397597.98 total 375.40 last 23.28) - open/close 8136 (time 1787397607.99 total 385.40 last 22.79) - open/close 8349 (time 1787397618.53 total 395.95 last 20.20) - open/close 8525 (time 1787397628.54 total 405.96 last 17.58) - open/close 8648 (time 1787397638.55 total 415.97 last 12.29) - open/close 8913 (time 1787397651.40 total 428.82 last 20.61) - open/close 9139 (time 1787397661.41 total 438.83 last 22.58) - open/close 9336 (time 1787397672.01 total 449.43 last 18.59) - open/close 9590 (time 1787397683.04 total 460.45 last 23.04) - open/close 9848 (time 1787397693.77 total 471.18 last 24.05) - open/close 10000 (time 1787397701.77 total 479.18 last 18.99) - open/close 10157 (time 1787397712.33 total 489.75 last 14.86) - open/close 10323 (time 1787397723.25 total 500.67 last 15.20) - open/close 10511 (time 1787397733.33 total 510.74 last 18.66) - open/close 10626 (time 1787397743.33 total 520.75 last 11.49) - open/close 10846 (time 1787397753.51 total 530.93 last 21.61) - open/close 11065 (time 1787397763.52 total 540.93 last 21.89) - open/close 11295 (time 1787397773.53 total 550.94 last 22.98) - open/close 11550 (time 1787397783.56 total 560.97 last 25.42) - open/close 11801 (time 1787397793.57 total 570.98 last 25.07) - open/close 12038 (time 1787397803.57 total 580.99 last 23.69) - open/close 12186 (time 1787397813.64 total 591.06 last 14.70) - open/close 12491 (time 1787397824.46 total 601.88 last 28.19) - open/close 12711 (time 1787397834.96 total 612.38 last 20.96) - open/close 12937 (time 1787397844.98 total 622.39 last 22.56) - open/close 13125 (time 1787397855.02 total 632.44 last 18.72) - open/close 13381 (time 1787397865.02 total 642.44 last 25.60) - open/close 13533 (time 1787397876.25 total 653.67 last 13.53) - open/close 13816 (time 1787397886.27 total 663.68 last 28.26) - open/close 14008 (time 1787397897.07 total 674.49 last 17.77) - open/close 14208 (time 1787397907.09 total 684.51 last 19.96) - open/close 14457 (time 1787397917.11 total 694.52 last 24.87) - open/close 14696 (time 1787397927.78 total 705.19 last 22.40) - open/close 14956 (time 1787397938.54 total 715.96 last 24.16) - open/close 15238 (time 1787397950.94 total 728.36 last 22.74) - open/close 15348 (time 1787397961.55 total 738.97 last 10.37) - open/close 15504 (time 1787397972.57 total 749.99 last 14.15) - open/close 15726 (time 1787397982.80 total 760.21 last 21.72) - open/close 16047 (time 1787397992.80 total 770.22 last 32.09) - open/close 16186 (time 1787398003.98 total 781.40 last 12.43) - open/close 16420 (time 1787398014.15 total 791.57 last 23.00) - open/close 16673 (time 1787398024.94 total 802.36 last 23.45) - open/close 16856 (time 1787398035.79 total 813.21 last 16.87) - open/close 17081 (time 1787398046.23 total 823.64 last 21.56) - open/close 17321 (time 1787398056.23 total 833.65 last 23.99) - open/close 17583 (time 1787398067.22 total 844.64 last 23.84) - open/close 17835 (time 1787398077.23 total 854.65 last 25.19) - open/close 18072 (time 1787398087.24 total 864.65 last 23.68) - open/close 18314 (time 1787398097.25 total 874.67 last 24.16) - open/close 18577 (time 1787398107.41 total 884.83 last 25.89) - open/close 18789 (time 1787398117.63 total 895.05 last 20.74) - open/close 19059 (time 1787398127.88 total 905.30 last 26.34) - open/close 19318 (time 1787398137.90 total 915.32 last 25.85) - open/close 19609 (time 1787398147.91 total 925.33 last 29.07) - open/close 19841 (time 1787398157.91 total 935.33 last 23.20) - open/close 20000 (time 1787398165.24 total 942.66 last 21.69) - open/close 20777 (time 1787398175.25 total 952.66 last 77.64) - open/close 21736 (time 1787398185.27 total 962.68 last 95.72) - open/close 22601 (time 1787398195.28 total 972.70 last 86.36) - open/close 23214 (time 1787398205.29 total 982.70 last 61.29) - open/close 23978 (time 1787398215.29 total 992.70 last 76.40) - open/close 24767 (time 1787398225.30 total 1002.72 last 78.76) - open/close 25779 (time 1787398235.30 total 1012.72 last 101.19) - open/close 26736 (time 1787398245.33 total 1022.74 last 95.47) - open/close 27647 (time 1787398255.33 total 1032.74 last 91.10) - open/close 28466 (time 1787398265.33 total 1042.74 last 81.90) - open/close 29289 (time 1787398275.33 total 1052.75 last 82.27) - open/close 30000 (time 1787398282.96 total 1060.38 last 93.16) - open/close 30928 (time 1787398292.97 total 1070.38 last 92.76) - open/close 31932 (time 1787398302.98 total 1080.40 last 100.25) - open/close 33016 (time 1787398312.99 total 1090.40 last 108.34) - open/close 33945 (time 1787398322.99 total 1100.41 last 92.86) - open/close 34899 (time 1787398332.99 total 1110.41 last 95.40) - open/close 35841 (time 1787398343.00 total 1120.42 last 94.12) - open/close 36717 (time 1787398353.00 total 1130.42 last 87.58) - open/close 37546 (time 1787398363.01 total 1140.43 last 82.82) - open/close 38224 (time 1787398373.02 total 1150.44 last 67.72) - open/close 39036 (time 1787398383.03 total 1160.45 last 81.12) - open/close 40000 (time 1787398391.82 total 1169.24 last 109.69) - open/close 41038 (time 1787398401.85 total 1179.27 last 103.52) - open/close 41994 (time 1787398411.85 total 1189.27 last 95.56) - open/close 42916 (time 1787398421.86 total 1199.28 last 92.13) - open/close 43799 (time 1787398431.86 total 1209.28 last 88.28) - open/close 44546 (time 1787398441.86 total 1219.28 last 74.69) - open/close 45260 (time 1787398451.87 total 1229.29 last 71.34) - open/close 46172 (time 1787398461.88 total 1239.30 last 91.11) - open/close 47112 (time 1787398471.89 total 1249.30 last 93.95) - open/close 48097 (time 1787398481.89 total 1259.31 last 98.46) - open/close 48874 (time 1787398491.89 total 1269.31 last 77.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1278.02 seconds: 38.83 ops/second - unlinked 0 (time 1787398505 ; total 0 ; last 0) - unlinked 10000 (time 1787398619 ; total 114 ; last 114) - unlinked 20000 (time 1787398731 ; total 226 ; last 112) - unlinked 30000 (time 1787398806 ; total 301 ; last 75) - unlinked 40000 (time 1787398879 ; total 374 ; last 73) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 437 seconds: 113.572083 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2