== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:45:29 (1769755529) start mds service on oleg415-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg415-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 159 (time 1769755610.72 total 10.88 last 14.61) - open/close 382 (time 1769755621.46 total 21.62 last 20.77) - open/close 506 (time 1769755631.79 total 31.95 last 12.00) - open/close 672 (time 1769755641.80 total 41.96 last 16.58) - open/close 828 (time 1769755653.74 total 53.90 last 13.07) - open/close 1003 (time 1769755664.03 total 64.19 last 17.01) - open/close 1238 (time 1769755674.15 total 74.31 last 23.22) - open/close 1395 (time 1769755685.21 total 85.37 last 14.19) - open/close 1493 (time 1769755695.34 total 95.50 last 9.68) - open/close 1626 (time 1769755705.35 total 105.51 last 13.28) - open/close 1817 (time 1769755716.30 total 116.47 last 17.43) - open/close 2072 (time 1769755726.31 total 126.47 last 25.49) - open/close 2214 (time 1769755736.67 total 136.84 last 13.70) - open/close 2349 (time 1769755746.69 total 146.85 last 13.48) - open/close 2501 (time 1769755756.70 total 156.86 last 15.19) - open/close 2598 (time 1769755767.77 total 167.94 last 8.76) - open/close 2716 (time 1769755777.80 total 177.96 last 11.77) - open/close 2787 (time 1769755787.80 total 187.96 last 7.10) - open/close 2974 (time 1769755797.81 total 197.97 last 18.68) - open/close 3039 (time 1769755807.84 total 208.00 last 6.48) - open/close 3182 (time 1769755818.22 total 218.38 last 13.78) - open/close 3411 (time 1769755828.41 total 228.57 last 22.48) - open/close 3566 (time 1769755838.90 total 239.06 last 14.77) - open/close 3680 (time 1769755849.11 total 249.27 last 11.17) - open/close 3920 (time 1769755859.14 total 259.30 last 23.93) - open/close 4025 (time 1769755869.19 total 269.35 last 10.45) - open/close 4160 (time 1769755879.19 total 279.35 last 13.50) - open/close 4403 (time 1769755889.76 total 289.92 last 22.98) - open/close 4495 (time 1769755899.76 total 299.93 last 9.20) - open/close 4687 (time 1769755911.06 total 311.22 last 17.00) - open/close 4817 (time 1769755923.26 total 323.42 last 10.65) - open/close 4978 (time 1769755933.30 total 333.46 last 16.04) - open/close 5109 (time 1769755944.18 total 344.34 last 12.04) - open/close 5295 (time 1769755954.78 total 354.94 last 17.55) - open/close 5453 (time 1769755964.94 total 365.10 last 15.55) - open/close 5565 (time 1769755975.65 total 375.81 last 10.46) - open/close 5649 (time 1769755986.08 total 386.24 last 8.05) - open/close 5824 (time 1769755996.09 total 396.25 last 17.49) - open/close 5921 (time 1769756007.77 total 407.93 last 8.30) - open/close 6031 (time 1769756017.78 total 417.94 last 10.99) - open/close 6169 (time 1769756028.51 total 428.68 last 12.85) - open/close 6318 (time 1769756038.53 total 438.69 last 14.88) - open/close 6465 (time 1769756048.54 total 448.70 last 14.69) - open/close 6621 (time 1769756059.13 total 459.29 last 14.73) - open/close 6736 (time 1769756069.14 total 469.30 last 11.48) - open/close 6896 (time 1769756082.79 total 482.95 last 11.73) - open/close 7074 (time 1769756092.79 total 492.95 last 17.80) - open/close 7224 (time 1769756103.79 total 503.96 last 13.63) - open/close 7441 (time 1769756114.60 total 514.76 last 20.08) - open/close 7534 (time 1769756124.62 total 524.78 last 9.28) - open/close 7749 (time 1769756135.65 total 535.81 last 19.50) - open/close 7993 (time 1769756147.75 total 547.91 last 20.16) - open/close 8166 (time 1769756158.31 total 558.48 last 16.37) - open/close 8260 (time 1769756169.12 total 569.28 last 8.70) - open/close 8427 (time 1769756179.12 total 579.28 last 16.70) - open/close 8521 (time 1769756189.79 total 589.95 last 8.81) - open/close 8676 (time 1769756204.96 total 605.13 last 10.22) - open/close 8723 (time 1769756215.71 total 615.87 last 4.37) - open/close 8880 (time 1769756225.73 total 625.89 last 15.68) - open/close 9084 (time 1769756235.76 total 635.93 last 20.32) - open/close 9280 (time 1769756245.77 total 645.93 last 19.59) - open/close 9392 (time 1769756256.23 total 656.39 last 10.70) - open/close 9529 (time 1769756267.29 total 667.45 last 12.39) - open/close 9728 (time 1769756277.29 total 677.45 last 19.90) - open/close 9857 (time 1769756287.30 total 687.46 last 12.89) - open/close 9979 (time 1769756297.45 total 697.61 last 12.02) - open/close 10113 (time 1769756307.47 total 707.63 last 13.37) - open/close 10228 (time 1769756317.92 total 718.08 last 11.00) - open/close 10351 (time 1769756328.15 total 728.32 last 12.02) - open/close 10488 (time 1769756338.18 total 738.35 last 13.66) - open/close 10600 (time 1769756348.77 total 748.94 last 10.58) - open/close 10746 (time 1769756359.57 total 759.73 last 13.52) - open/close 10874 (time 1769756369.59 total 769.75 last 12.78) - open/close 10968 (time 1769756380.60 total 780.76 last 8.54) - open/close 11061 (time 1769756391.60 total 791.76 last 8.45) - open/close 11188 (time 1769756401.66 total 801.82 last 12.62) - open/close 11313 (time 1769756411.66 total 811.83 last 12.50) - open/close 11466 (time 1769756422.62 total 822.79 last 13.96) - open/close 11571 (time 1769756432.64 total 832.80 last 10.48) - open/close 11707 (time 1769756443.43 total 843.60 last 12.60) - open/close 11851 (time 1769756453.45 total 853.61 last 14.37) - open/close 12074 (time 1769756464.15 total 864.31 last 20.84) - open/close 12204 (time 1769756474.81 total 874.98 last 12.19) - open/close 12312 (time 1769756484.84 total 885.00 last 10.78) - open/close 12468 (time 1769756495.71 total 895.87 last 14.35) - open/close 12561 (time 1769756505.72 total 905.88 last 9.29) - open/close 12586 (time 1769756516.56 total 916.73 last 2.31) - open/close 12683 (time 1769756526.57 total 926.73 last 9.69) - open/close 12817 (time 1769756536.58 total 936.74 last 13.39) - open/close 12953 (time 1769756546.58 total 946.75 last 13.59) - open/close 13087 (time 1769756557.15 total 957.31 last 12.68) - open/close 13288 (time 1769756567.51 total 967.67 last 19.40) - open/close 13417 (time 1769756577.92 total 978.08 last 12.39) - open/close 13539 (time 1769756588.47 total 988.63 last 11.57) - open/close 13690 (time 1769756598.49 total 998.65 last 15.07) - open/close 13809 (time 1769756608.74 total 1008.90 last 11.60) - open/close 14038 (time 1769756619.36 total 1019.53 last 21.56) - open/close 14062 (time 1769756631.44 total 1031.60 last 1.99) - open/close 14218 (time 1769756642.53 total 1042.70 last 14.06) - open/close 14342 (time 1769756653.31 total 1053.47 last 11.51) - open/close 14476 (time 1769756663.72 total 1063.88 last 12.87) - open/close 14698 (time 1769756673.72 total 1073.89 last 22.19) - open/close 14801 (time 1769756683.73 total 1083.89 last 10.29) - open/close 14872 (time 1769756693.90 total 1094.06 last 6.98) - open/close 15083 (time 1769756704.80 total 1104.97 last 19.34) - open/close 15242 (time 1769756714.82 total 1114.98 last 15.87) - open/close 15432 (time 1769756731.03 total 1131.20 last 11.72) - open/close 15954 (time 1769756741.04 total 1141.20 last 52.16) - open/close 16633 (time 1769756751.04 total 1151.21 last 67.88) - open/close 17178 (time 1769756761.05 total 1161.21 last 54.50) - open/close 17632 (time 1769756771.05 total 1171.21 last 45.39) - open/close 18266 (time 1769756781.06 total 1181.22 last 63.33) - open/close 18922 (time 1769756791.06 total 1191.22 last 65.59) - open/close 19609 (time 1769756801.07 total 1201.23 last 68.62) - open/close 20000 (time 1769756806.17 total 1206.33 last 76.72) - open/close 20792 (time 1769756816.17 total 1216.33 last 79.19) - open/close 21498 (time 1769756826.21 total 1226.37 last 70.31) - open/close 22038 (time 1769756836.23 total 1236.39 last 53.92) - open/close 22603 (time 1769756846.24 total 1246.40 last 56.44) - open/close 23260 (time 1769756856.24 total 1256.40 last 65.68) - open/close 23804 (time 1769756866.25 total 1266.41 last 54.35) - open/close 24379 (time 1769756876.25 total 1276.41 last 57.49) - open/close 25064 (time 1769756886.25 total 1286.41 last 68.50) - open/close 25833 (time 1769756896.26 total 1296.42 last 76.85) - open/close 26570 (time 1769756906.26 total 1306.42 last 73.69) - open/close 27231 (time 1769756916.27 total 1316.43 last 66.04) - open/close 27816 (time 1769756926.27 total 1326.43 last 58.49) - open/close 28424 (time 1769756936.28 total 1336.44 last 60.75) - open/close 29178 (time 1769756946.28 total 1346.44 last 75.39) - open/close 29700 (time 1769756956.28 total 1356.45 last 52.18) - open/close 30000 (time 1769756961.56 total 1361.72 last 56.91) - open/close 30445 (time 1769756971.57 total 1371.73 last 44.42) - open/close 31110 (time 1769756981.58 total 1381.75 last 66.43) - open/close 31828 (time 1769756991.59 total 1391.75 last 71.77) - open/close 32511 (time 1769757001.59 total 1401.75 last 68.28) - open/close 33247 (time 1769757011.59 total 1411.75 last 73.59) - open/close 33915 (time 1769757021.60 total 1421.76 last 66.76) - open/close 34526 (time 1769757031.61 total 1431.77 last 61.05) - open/close 35161 (time 1769757041.61 total 1441.77 last 63.46) - open/close 35797 (time 1769757051.62 total 1451.78 last 63.58) - open/close 36443 (time 1769757061.62 total 1461.78 last 64.58) - open/close 37140 (time 1769757071.62 total 1471.78 last 69.68) - open/close 37799 (time 1769757081.66 total 1481.82 last 65.64) - open/close 38424 (time 1769757091.67 total 1491.83 last 62.45) - open/close 39040 (time 1769757101.68 total 1501.84 last 61.56) - open/close 39667 (time 1769757111.68 total 1511.84 last 62.67) - open/close 40000 (time 1769757116.52 total 1516.68 last 68.77) - open/close 40713 (time 1769757126.52 total 1526.69 last 71.29) - open/close 41457 (time 1769757136.53 total 1536.69 last 74.35) - open/close 42088 (time 1769757146.55 total 1546.71 last 62.98) - open/close 42616 (time 1769757156.56 total 1556.73 last 52.73) - open/close 43281 (time 1769757166.58 total 1566.74 last 66.41) - open/close 44087 (time 1769757176.59 total 1576.75 last 80.50) - open/close 44828 (time 1769757186.59 total 1586.76 last 74.07) - open/close 45497 (time 1769757196.60 total 1596.76 last 66.84) - open/close 46207 (time 1769757206.62 total 1606.78 last 70.91) - open/close 46879 (time 1769757216.63 total 1616.79 last 67.13) - open/close 47657 (time 1769757226.63 total 1626.79 last 77.78) - open/close 48311 (time 1769757236.65 total 1636.81 last 65.25) - open/close 49071 (time 1769757246.66 total 1646.82 last 75.94) - open/close 49833 (time 1769757256.66 total 1656.82 last 76.18) total: 50000 open/close in 1659.22 seconds: 30.13 ops/second - unlinked 0 (time 1769757264 ; total 1 ; last 1) - unlinked 10000 (time 1769757413 ; total 150 ; last 149) - unlinked 20000 (time 1769757564 ; total 301 ; last 151) - unlinked 30000 (time 1769757706 ; total 443 ; last 142) - unlinked 40000 (time 1769757842 ; total 579 ; last 136) total: 50000 unlinks in 703 seconds: 71.123756 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local oleg415-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg415-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2