== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:39:22 (1787387962) start mds service on oleg253-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing load_modules_local oleg253-server: Loading modules from /home/green/git/lustre-release/lustre oleg253-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg253-server: detected 4 online CPUs by sysfs oleg253-server: MODOPTS_LIBCFS= oleg253-server: Force libcfs to create 2 CPU partitions oleg253-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg253-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg253-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg253-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg253-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1787388081.51 total 10.30 last 22.24) - open/close 488 (time 1787388091.53 total 20.32 last 25.83) - open/close 559 (time 1787388102.50 total 31.29 last 6.47) - open/close 700 (time 1787388112.86 total 41.65 last 13.61) - open/close 816 (time 1787388123.31 total 52.10 last 11.10) - open/close 984 (time 1787388133.45 total 62.24 last 16.58) - open/close 1237 (time 1787388144.37 total 73.16 last 23.17) - open/close 1491 (time 1787388154.39 total 83.17 last 25.36) - open/close 1789 (time 1787388164.89 total 93.67 last 28.38) - open/close 2045 (time 1787388175.54 total 104.33 last 24.02) - open/close 2291 (time 1787388185.56 total 114.35 last 24.57) - open/close 2430 (time 1787388195.69 total 124.48 last 13.72) - open/close 2643 (time 1787388206.31 total 135.10 last 20.04) - open/close 2940 (time 1787388216.32 total 145.11 last 29.69) - open/close 3191 (time 1787388226.52 total 155.31 last 24.59) - open/close 3438 (time 1787388236.53 total 165.32 last 24.70) - open/close 3579 (time 1787388247.29 total 176.08 last 13.10) - open/close 3731 (time 1787388257.95 total 186.74 last 14.26) - open/close 3902 (time 1787388268.79 total 197.58 last 15.78) - open/close 4113 (time 1787388281.90 total 210.69 last 16.09) - open/close 4283 (time 1787388293.70 total 222.49 last 14.40) - open/close 4512 (time 1787388304.66 total 233.45 last 20.89) - open/close 4724 (time 1787388315.36 total 244.15 last 19.82) - open/close 4867 (time 1787388325.37 total 254.16 last 14.28) - open/close 5064 (time 1787388335.45 total 264.24 last 19.54) - open/close 5198 (time 1787388346.04 total 274.83 last 12.65) - open/close 5334 (time 1787388356.06 total 284.85 last 13.58) - open/close 5405 (time 1787388366.69 total 295.48 last 6.68) - open/close 5606 (time 1787388377.67 total 306.46 last 18.31) - open/close 5865 (time 1787388387.68 total 316.47 last 25.87) - open/close 6090 (time 1787388398.12 total 326.91 last 21.55) - open/close 6299 (time 1787388408.24 total 337.03 last 20.65) - open/close 6521 (time 1787388418.25 total 347.04 last 22.19) - open/close 6682 (time 1787388429.11 total 357.90 last 14.82) - open/close 6906 (time 1787388439.12 total 367.91 last 22.37) - open/close 7039 (time 1787388449.95 total 378.74 last 12.28) - open/close 7226 (time 1787388462.63 total 391.42 last 14.75) - open/close 7384 (time 1787388472.63 total 401.42 last 15.80) - open/close 7639 (time 1787388483.56 total 412.35 last 23.32) - open/close 7923 (time 1787388493.92 total 422.71 last 27.41) - open/close 8062 (time 1787388503.95 total 432.74 last 13.87) - open/close 8231 (time 1787388514.60 total 443.39 last 15.87) - open/close 8315 (time 1787388525.67 total 454.46 last 7.59) - open/close 8430 (time 1787388535.67 total 464.46 last 11.50) - open/close 8613 (time 1787388545.69 total 474.48 last 18.28) - open/close 8934 (time 1787388555.70 total 484.49 last 32.06) - open/close 9131 (time 1787388567.84 total 496.63 last 16.22) - open/close 9312 (time 1787388577.86 total 506.65 last 18.07) - open/close 9436 (time 1787388588.77 total 517.56 last 11.36) - open/close 9691 (time 1787388599.40 total 528.19 last 23.99) - open/close 9832 (time 1787388610.48 total 539.27 last 12.72) - open/close 10000 (time 1787388620.02 total 548.81 last 17.61) - open/close 10298 (time 1787388630.83 total 559.61 last 27.58) - open/close 10476 (time 1787388641.31 total 570.10 last 16.98) - open/close 10648 (time 1787388651.42 total 580.21 last 17.01) - open/close 10774 (time 1787388661.53 total 590.32 last 12.46) - open/close 10979 (time 1787388671.56 total 600.35 last 20.44) - open/close 11203 (time 1787388682.48 total 611.27 last 20.52) - open/close 11488 (time 1787388693.22 total 622.01 last 26.54) - open/close 11671 (time 1787388703.22 total 632.01 last 18.29) - open/close 11749 (time 1787388713.32 total 642.11 last 7.72) - open/close 11983 (time 1787388723.33 total 652.12 last 23.40) - open/close 12194 (time 1787388735.95 total 664.74 last 16.72) - open/close 12370 (time 1787388747.11 total 675.90 last 15.77) - open/close 12504 (time 1787388757.20 total 685.99 last 13.28) - open/close 12663 (time 1787388767.65 total 696.44 last 15.21) - open/close 12854 (time 1787388780.07 total 708.86 last 15.38) - open/close 12983 (time 1787388790.07 total 718.86 last 12.90) - open/close 13116 (time 1787388800.27 total 729.05 last 13.05) - open/close 13383 (time 1787388810.74 total 739.53 last 25.49) - open/close 13465 (time 1787388823.08 total 751.87 last 6.65) - open/close 13679 (time 1787388833.54 total 762.32 last 20.47) - open/close 13773 (time 1787388843.70 total 772.49 last 9.25) - open/close 14098 (time 1787388854.44 total 783.23 last 30.26) - open/close 14304 (time 1787388864.44 total 793.23 last 20.60) - open/close 14489 (time 1787388874.44 total 803.23 last 18.49) - open/close 14676 (time 1787388885.28 total 814.07 last 17.25) - open/close 14840 (time 1787388895.29 total 824.08 last 16.39) - open/close 15046 (time 1787388905.83 total 834.62 last 19.55) - open/close 15121 (time 1787388916.13 total 844.92 last 7.28) - open/close 15313 (time 1787388926.13 total 854.92 last 19.19) - open/close 15465 (time 1787388936.62 total 865.41 last 14.50) - open/close 15643 (time 1787388946.62 total 875.41 last 17.80) - open/close 15723 (time 1787388956.74 total 885.53 last 7.91) - open/close 15896 (time 1787388966.74 total 895.53 last 17.30) - open/close 16133 (time 1787388977.15 total 905.94 last 22.77) - open/close 16270 (time 1787388987.30 total 916.09 last 13.50) - open/close 16448 (time 1787388998.11 total 926.90 last 16.46) - open/close 16569 (time 1787389008.16 total 936.95 last 12.04) - open/close 16691 (time 1787389018.79 total 947.58 last 11.47) - open/close 16778 (time 1787389028.96 total 957.75 last 8.56) - open/close 16927 (time 1787389041.72 total 970.51 last 11.68) - open/close 17100 (time 1787389052.64 total 981.43 last 15.83) - open/close 17297 (time 1787389062.96 total 991.75 last 19.09) - open/close 17412 (time 1787389072.99 total 1001.78 last 11.47) - open/close 17624 (time 1787389083.00 total 1011.79 last 21.18) - open/close 17792 (time 1787389093.00 total 1021.79 last 16.79) - open/close 17934 (time 1787389103.52 total 1032.31 last 13.50) - open/close 18122 (time 1787389113.54 total 1042.33 last 18.76) - open/close 18253 (time 1787389123.69 total 1052.48 last 12.91) - open/close 18900 (time 1787389133.70 total 1062.49 last 64.64) - open/close 19595 (time 1787389143.71 total 1072.50 last 69.39) - open/close 20000 (time 1787389148.84 total 1077.63 last 79.00) - open/close 20853 (time 1787389158.84 total 1087.63 last 85.26) - open/close 21739 (time 1787389168.85 total 1097.63 last 88.59) - open/close 22747 (time 1787389178.85 total 1107.64 last 100.76) - open/close 23549 (time 1787389188.85 total 1117.64 last 80.18) - open/close 24448 (time 1787389198.86 total 1127.65 last 89.84) - open/close 25346 (time 1787389208.87 total 1137.66 last 89.68) - open/close 26113 (time 1787389218.87 total 1147.66 last 76.68) - open/close 26878 (time 1787389228.88 total 1157.67 last 76.45) - open/close 27719 (time 1787389238.89 total 1167.67 last 84.06) - open/close 28532 (time 1787389248.89 total 1177.68 last 81.25) - open/close 29328 (time 1787389258.91 total 1187.70 last 79.47) - open/close 30000 (time 1787389267.79 total 1196.58 last 75.68) - open/close 30878 (time 1787389277.79 total 1206.58 last 87.76) - open/close 31803 (time 1787389287.79 total 1216.58 last 92.50) - open/close 32573 (time 1787389297.80 total 1226.59 last 76.96) - open/close 33268 (time 1787389307.80 total 1236.59 last 69.44) - open/close 34153 (time 1787389317.81 total 1246.60 last 88.46) - open/close 34928 (time 1787389327.83 total 1256.61 last 77.38) - open/close 35675 (time 1787389337.88 total 1266.66 last 74.33) - open/close 36246 (time 1787389347.91 total 1276.70 last 56.90) - open/close 37017 (time 1787389357.94 total 1286.73 last 76.88) - open/close 37720 (time 1787389367.97 total 1296.76 last 70.08) - open/close 38420 (time 1787389378.01 total 1306.80 last 69.72) - open/close 39169 (time 1787389388.02 total 1316.81 last 74.82) - open/close 39878 (time 1787389398.04 total 1326.83 last 70.76) - open/close 40000 (time 1787389400.11 total 1328.89 last 59.09) - open/close 40722 (time 1787389410.11 total 1338.90 last 72.15) - open/close 41557 (time 1787389420.12 total 1348.91 last 83.41) - open/close 42422 (time 1787389430.13 total 1358.92 last 86.44) - open/close 43376 (time 1787389440.15 total 1368.93 last 95.26) - open/close 44103 (time 1787389450.15 total 1378.93 last 72.70) - open/close 45021 (time 1787389460.15 total 1388.94 last 91.74) - open/close 45824 (time 1787389470.15 total 1398.94 last 80.29) - open/close 46639 (time 1787389480.16 total 1408.95 last 81.46) - open/close 47427 (time 1787389490.17 total 1418.96 last 78.73) - open/close 48094 (time 1787389500.17 total 1428.96 last 66.66) - open/close 48883 (time 1787389510.19 total 1438.98 last 78.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1448.85 seconds: 34.26 ops/second - unlinked 0 (time 1787389525 ; total 0 ; last 0) - unlinked 10000 (time 1787389664 ; total 139 ; last 139) - unlinked 20000 (time 1787389800 ; total 275 ; last 136) - unlinked 30000 (time 1787389952 ; total 427 ; last 152) - unlinked 40000 (time 1787390084 ; total 559 ; last 132) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 669 seconds: 74.186844 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg253-server unloading modules via unload_modules_local on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local oleg253-server: modules unloaded. pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2