== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:28:06 (1769059686) start mds service on oleg451-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1769059781.97 total 10.01 last 22.89) - open/close 479 (time 1769059792.63 total 20.66 last 23.46) - open/close 705 (time 1769059803.19 total 31.23 last 21.40) - open/close 854 (time 1769059813.33 total 41.36 last 14.70) - open/close 1014 (time 1769059823.35 total 51.39 last 15.96) - open/close 1174 (time 1769059833.36 total 61.40 last 15.99) - open/close 1311 (time 1769059843.36 total 71.40 last 13.70) - open/close 1413 (time 1769059853.97 total 82.01 last 9.61) - open/close 1515 (time 1769059863.98 total 92.02 last 10.19) - open/close 1749 (time 1769059874.89 total 102.93 last 21.44) - open/close 1942 (time 1769059884.91 total 112.94 last 19.27) - open/close 2106 (time 1769059895.88 total 123.92 last 14.95) - open/close 2221 (time 1769059906.46 total 134.49 last 10.87) - open/close 2450 (time 1769059917.01 total 145.05 last 21.69) - open/close 2603 (time 1769059927.59 total 155.62 last 14.47) - open/close 2769 (time 1769059937.87 total 165.91 last 16.14) - open/close 3040 (time 1769059948.66 total 176.70 last 25.11) - open/close 3223 (time 1769059958.69 total 186.72 last 18.25) - open/close 3446 (time 1769059969.40 total 197.44 last 20.81) - open/close 3661 (time 1769059979.84 total 207.88 last 20.59) - open/close 3868 (time 1769059990.39 total 218.43 last 19.62) - open/close 4055 (time 1769060000.42 total 228.46 last 18.64) - open/close 4295 (time 1769060010.89 total 238.92 last 22.94) - open/close 4575 (time 1769060021.51 total 249.55 last 26.35) - open/close 4848 (time 1769060032.54 total 260.57 last 24.76) - open/close 5174 (time 1769060042.54 total 270.58 last 32.58) - open/close 5431 (time 1769060053.25 total 281.29 last 24.01) - open/close 5681 (time 1769060063.42 total 291.46 last 24.58) - open/close 5925 (time 1769060073.78 total 301.82 last 23.55) - open/close 6130 (time 1769060084.82 total 312.86 last 18.57) - open/close 6321 (time 1769060095.81 total 323.85 last 17.38) - open/close 6462 (time 1769060106.77 total 334.81 last 12.86) - open/close 6637 (time 1769060118.47 total 346.50 last 14.96) - open/close 6866 (time 1769060128.78 total 356.82 last 22.20) - open/close 7158 (time 1769060138.79 total 366.82 last 29.18) - open/close 7244 (time 1769060149.25 total 377.29 last 8.22) - open/close 7435 (time 1769060159.49 total 387.52 last 18.66) - open/close 7698 (time 1769060170.36 total 398.40 last 24.18) - open/close 7901 (time 1769060180.37 total 408.41 last 20.28) - open/close 8168 (time 1769060190.53 total 418.57 last 26.28) - open/close 8400 (time 1769060204.62 total 432.66 last 16.47) - open/close 8647 (time 1769060215.06 total 443.09 last 23.67) - open/close 8888 (time 1769060227.06 total 455.10 last 20.07) - open/close 9089 (time 1769060237.31 total 465.35 last 19.62) - open/close 9321 (time 1769060247.69 total 475.72 last 22.36) - open/close 9529 (time 1769060257.70 total 485.73 last 20.78) - open/close 9705 (time 1769060268.44 total 496.48 last 16.38) - open/close 9904 (time 1769060278.48 total 506.52 last 19.81) - open/close 10000 (time 1769060282.19 total 510.23 last 25.89) - open/close 10275 (time 1769060292.21 total 520.24 last 27.46) - open/close 10551 (time 1769060303.25 total 531.29 last 24.99) - open/close 10822 (time 1769060314.02 total 542.05 last 25.18) - open/close 11055 (time 1769060324.63 total 552.66 last 21.96) - open/close 11271 (time 1769060334.65 total 562.68 last 21.56) - open/close 11494 (time 1769060344.92 total 572.95 last 21.71) - open/close 11722 (time 1769060355.55 total 583.58 last 21.45) - open/close 11952 (time 1769060365.83 total 593.87 last 22.36) - open/close 12222 (time 1769060375.85 total 603.89 last 26.94) - open/close 12458 (time 1769060385.87 total 613.90 last 23.57) - open/close 12690 (time 1769060396.03 total 624.06 last 22.83) - open/close 12872 (time 1769060406.07 total 634.11 last 18.12) - open/close 13106 (time 1769060416.08 total 644.11 last 23.38) - open/close 13397 (time 1769060426.08 total 654.12 last 29.09) - open/close 13629 (time 1769060436.37 total 664.40 last 22.55) - open/close 13856 (time 1769060446.45 total 674.49 last 22.51) - open/close 14157 (time 1769060456.63 total 684.66 last 29.59) - open/close 14446 (time 1769060466.65 total 694.68 last 28.84) - open/close 14695 (time 1769060476.65 total 704.69 last 24.89) - open/close 14988 (time 1769060486.67 total 714.71 last 29.24) - open/close 15233 (time 1769060496.68 total 724.71 last 24.49) - open/close 15473 (time 1769060506.70 total 734.73 last 23.95) - open/close 15724 (time 1769060516.72 total 744.75 last 25.06) - open/close 15975 (time 1769060527.58 total 755.61 last 23.11) - open/close 16218 (time 1769060538.24 total 766.28 last 22.79) - open/close 16535 (time 1769060549.24 total 777.28 last 28.81) - open/close 16784 (time 1769060559.50 total 787.53 last 24.28) - open/close 17000 (time 1769060569.50 total 797.54 last 21.59) - open/close 17251 (time 1769060579.51 total 807.55 last 25.08) - open/close 17502 (time 1769060589.57 total 817.61 last 24.95) - open/close 17729 (time 1769060600.44 total 828.48 last 20.88) - open/close 18044 (time 1769060610.45 total 838.49 last 31.47) - open/close 18271 (time 1769060621.00 total 849.04 last 21.52) - open/close 18572 (time 1769060631.69 total 859.73 last 28.16) - open/close 18727 (time 1769060641.72 total 869.75 last 15.46) - open/close 19038 (time 1769060651.72 total 879.76 last 31.08) - open/close 19237 (time 1769060662.02 total 890.06 last 19.32) - open/close 19429 (time 1769060672.06 total 900.09 last 19.14) - open/close 19660 (time 1769060682.62 total 910.66 last 21.86) - open/close 19880 (time 1769060692.82 total 920.86 last 21.57) - open/close 20000 (time 1769060698.81 total 926.85 last 20.02) - open/close 21034 (time 1769060708.81 total 936.85 last 103.37) - open/close 22194 (time 1769060718.82 total 946.86 last 115.95) - open/close 22835 (time 1769060728.84 total 956.87 last 63.99) - open/close 23334 (time 1769060738.86 total 966.90 last 49.77) - open/close 23763 (time 1769060748.87 total 976.90 last 42.89) - open/close 24092 (time 1769060758.87 total 986.91 last 32.88) - open/close 24679 (time 1769060768.88 total 996.92 last 58.65) - open/close 25273 (time 1769060778.90 total 1006.93 last 59.29) - open/close 26068 (time 1769060788.90 total 1016.94 last 79.47) - open/close 26954 (time 1769060798.93 total 1026.96 last 88.37) - open/close 27837 (time 1769060808.94 total 1036.98 last 88.17) - open/close 28473 (time 1769060818.95 total 1046.99 last 63.55) - open/close 29281 (time 1769060828.96 total 1056.99 last 80.75) - open/close 30000 (time 1769060836.48 total 1064.52 last 95.53) - open/close 30999 (time 1769060846.48 total 1074.52 last 99.90) - open/close 31770 (time 1769060856.48 total 1084.52 last 77.09) - open/close 32424 (time 1769060866.49 total 1094.53 last 65.35) - open/close 33043 (time 1769060876.49 total 1104.53 last 61.90) - open/close 33625 (time 1769060886.50 total 1114.53 last 58.17) - open/close 34457 (time 1769060896.50 total 1124.54 last 83.16) - open/close 35305 (time 1769060906.51 total 1134.55 last 84.71) - open/close 36141 (time 1769060916.52 total 1144.55 last 83.56) - open/close 36953 (time 1769060926.52 total 1154.56 last 81.14) - open/close 37743 (time 1769060936.54 total 1164.58 last 78.85) - open/close 38564 (time 1769060946.55 total 1174.59 last 82.05) - open/close 39273 (time 1769060956.55 total 1184.59 last 70.89) - open/close 40000 (time 1769060964.19 total 1192.23 last 95.10) - open/close 40755 (time 1769060974.20 total 1202.23 last 75.49) - open/close 41588 (time 1769060984.20 total 1212.23 last 83.30) - open/close 42421 (time 1769060994.20 total 1222.24 last 83.24) - open/close 43297 (time 1769061004.21 total 1232.24 last 87.56) - open/close 44186 (time 1769061014.22 total 1242.25 last 88.82) - open/close 44953 (time 1769061024.23 total 1252.27 last 76.60) - open/close 45743 (time 1769061034.24 total 1262.27 last 78.94) - open/close 46589 (time 1769061044.24 total 1272.28 last 84.59) - open/close 47367 (time 1769061054.24 total 1282.28 last 77.79) - open/close 48134 (time 1769061064.26 total 1292.30 last 76.55) - open/close 48888 (time 1769061074.27 total 1302.31 last 75.29) - open/close 49613 (time 1769061084.28 total 1312.31 last 72.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1312.53 seconds: 37.81 ops/second - unlinked 0 (time 1769061090 ; total 0 ; last 0) - unlinked 10000 (time 1769061224 ; total 134 ; last 134) - unlinked 20000 (time 1769061342 ; total 252 ; last 118) - unlinked 30000 (time 1769061449 ; total 359 ; last 107) - unlinked 40000 (time 1769061521 ; total 431 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 487 seconds: 101.913757 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg451-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2