== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:34:44 (1786800884) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg321-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 206 (time 1786800983.06 total 10.02 last 20.57) - open/close 416 (time 1786800993.06 total 20.02 last 21.00) - open/close 577 (time 1786801003.07 total 30.03 last 16.09) - open/close 746 (time 1786801014.85 total 41.81 last 14.34) - open/close 874 (time 1786801026.89 total 53.85 last 10.64) - open/close 1070 (time 1786801036.90 total 63.85 last 19.58) - open/close 1237 (time 1786801047.43 total 74.39 last 15.85) - open/close 1453 (time 1786801057.43 total 84.39 last 21.59) - open/close 1657 (time 1786801067.83 total 94.78 last 19.63) - open/close 1896 (time 1786801078.46 total 105.42 last 22.47) - open/close 2118 (time 1786801088.76 total 115.72 last 21.55) - open/close 2328 (time 1786801099.09 total 126.05 last 20.32) - open/close 2630 (time 1786801109.57 total 136.53 last 28.82) - open/close 2882 (time 1786801121.46 total 148.42 last 21.20) - open/close 3093 (time 1786801131.46 total 158.42 last 21.10) - open/close 3351 (time 1786801141.47 total 168.43 last 25.77) - open/close 3535 (time 1786801151.51 total 178.46 last 18.33) - open/close 3716 (time 1786801162.07 total 189.03 last 17.13) - open/close 3933 (time 1786801172.73 total 199.68 last 20.37) - open/close 4221 (time 1786801183.69 total 210.65 last 26.27) - open/close 4411 (time 1786801193.69 total 220.65 last 18.99) - open/close 4573 (time 1786801204.16 total 231.11 last 15.49) - open/close 4733 (time 1786801214.18 total 241.14 last 15.96) - open/close 4938 (time 1786801224.77 total 251.72 last 19.36) - open/close 5178 (time 1786801234.77 total 261.73 last 23.99) - open/close 5407 (time 1786801245.36 total 272.32 last 21.62) - open/close 5662 (time 1786801255.37 total 282.33 last 25.48) - open/close 5880 (time 1786801267.28 total 294.23 last 18.31) - open/close 6103 (time 1786801277.30 total 304.25 last 22.25) - open/close 6322 (time 1786801287.33 total 314.28 last 21.84) - open/close 6585 (time 1786801297.91 total 324.87 last 24.85) - open/close 6830 (time 1786801308.27 total 335.23 last 23.64) - open/close 7110 (time 1786801319.03 total 345.99 last 26.04) - open/close 7391 (time 1786801329.87 total 356.82 last 25.93) - open/close 7680 (time 1786801340.36 total 367.31 last 27.55) - open/close 7919 (time 1786801350.37 total 377.33 last 23.87) - open/close 8157 (time 1786801361.08 total 388.04 last 22.21) - open/close 8376 (time 1786801371.09 total 398.05 last 21.88) - open/close 8563 (time 1786801381.82 total 408.77 last 17.44) - open/close 8710 (time 1786801391.82 total 418.78 last 14.69) - open/close 8872 (time 1786801402.00 total 428.95 last 15.92) - open/close 9029 (time 1786801412.29 total 439.25 last 15.25) - open/close 9182 (time 1786801422.90 total 449.86 last 14.41) - open/close 9361 (time 1786801433.37 total 460.33 last 17.09) - open/close 9586 (time 1786801443.77 total 470.73 last 21.64) - open/close 9837 (time 1786801453.77 total 480.73 last 25.10) - open/close 10000 (time 1786801460.65 total 487.61 last 23.70) - open/close 10285 (time 1786801471.17 total 498.13 last 27.10) - open/close 10439 (time 1786801481.76 total 508.72 last 14.54) - open/close 10575 (time 1786801492.74 total 519.70 last 12.39) - open/close 10846 (time 1786801503.69 total 530.65 last 24.74) - open/close 11098 (time 1786801514.10 total 541.06 last 24.21) - open/close 11314 (time 1786801524.12 total 551.07 last 21.57) - open/close 11559 (time 1786801534.80 total 561.76 last 22.93) - open/close 11759 (time 1786801544.81 total 571.76 last 19.99) - open/close 11962 (time 1786801554.83 total 581.78 last 20.26) - open/close 12093 (time 1786801564.84 total 591.80 last 13.08) - open/close 12246 (time 1786801575.61 total 602.57 last 14.20) - open/close 12514 (time 1786801586.68 total 613.64 last 24.22) - open/close 12690 (time 1786801597.38 total 624.34 last 16.44) - open/close 12857 (time 1786801607.39 total 634.35 last 16.69) - open/close 13095 (time 1786801617.60 total 644.55 last 23.32) - open/close 13321 (time 1786801627.64 total 654.60 last 22.49) - open/close 13554 (time 1786801637.94 total 664.90 last 22.63) - open/close 13814 (time 1786801647.94 total 674.90 last 26.00) - open/close 13985 (time 1786801659.06 total 686.01 last 15.39) - open/close 14204 (time 1786801669.99 total 696.95 last 20.03) - open/close 14455 (time 1786801680.12 total 707.07 last 24.79) - open/close 14722 (time 1786801690.12 total 717.08 last 26.70) - open/close 14960 (time 1786801700.13 total 727.09 last 23.77) - open/close 15193 (time 1786801710.73 total 737.68 last 21.99) - open/close 15422 (time 1786801720.74 total 747.69 last 22.87) - open/close 15769 (time 1786801730.74 total 757.70 last 34.68) - open/close 15882 (time 1786801741.14 total 768.10 last 10.87) - open/close 16138 (time 1786801751.60 total 778.56 last 24.47) - open/close 16403 (time 1786801761.65 total 788.61 last 26.35) - open/close 16594 (time 1786801771.96 total 798.92 last 18.53) - open/close 16858 (time 1786801781.98 total 808.94 last 26.35) - open/close 17091 (time 1786801792.46 total 819.42 last 22.23) - open/close 17303 (time 1786801802.55 total 829.51 last 21.00) - open/close 17485 (time 1786801813.24 total 840.20 last 17.03) - open/close 17635 (time 1786801825.21 total 852.17 last 12.54) - open/close 17823 (time 1786801835.21 total 862.17 last 18.80) - open/close 17927 (time 1786801846.03 total 872.99 last 9.61) - open/close 18177 (time 1786801856.05 total 883.00 last 24.96) - open/close 18283 (time 1786801866.18 total 893.13 last 10.46) - open/close 18442 (time 1786801876.99 total 903.95 last 14.70) - open/close 18704 (time 1786801887.01 total 913.97 last 26.14) - open/close 18804 (time 1786801897.32 total 924.28 last 9.70) - open/close 18990 (time 1786801908.17 total 935.12 last 17.15) - open/close 19191 (time 1786801918.22 total 945.17 last 20.00) - open/close 19930 (time 1786801928.22 total 955.17 last 73.89) - open/close 20506 (time 1786801938.22 total 965.18 last 57.58) - open/close 21358 (time 1786801948.23 total 975.18 last 85.16) - open/close 22097 (time 1786801958.23 total 985.18 last 73.90) - open/close 22659 (time 1786801968.23 total 995.19 last 56.16) - open/close 23291 (time 1786801978.24 total 1005.20 last 63.16) - open/close 23953 (time 1786801988.24 total 1015.20 last 66.17) - open/close 24788 (time 1786801998.26 total 1025.22 last 83.37) - open/close 25652 (time 1786802008.26 total 1035.22 last 86.39) - open/close 26500 (time 1786802018.26 total 1045.22 last 84.76) - open/close 27287 (time 1786802028.27 total 1055.23 last 78.64) - open/close 28011 (time 1786802038.28 total 1065.24 last 72.36) - open/close 28813 (time 1786802048.29 total 1075.24 last 80.13) - open/close 29604 (time 1786802058.32 total 1085.27 last 78.87) - open/close 30000 (time 1786802063.04 total 1090.00 last 83.77) - open/close 30720 (time 1786802073.05 total 1100.01 last 71.95) - open/close 31565 (time 1786802083.08 total 1110.04 last 84.24) - open/close 32273 (time 1786802093.09 total 1120.05 last 70.75) - open/close 33104 (time 1786802103.09 total 1130.05 last 83.06) - open/close 33830 (time 1786802113.10 total 1140.06 last 72.55) - open/close 34615 (time 1786802123.10 total 1150.06 last 78.49) - open/close 35448 (time 1786802133.10 total 1160.06 last 83.30) - open/close 36282 (time 1786802143.11 total 1170.07 last 83.33) - open/close 36976 (time 1786802153.12 total 1180.08 last 69.33) - open/close 37700 (time 1786802163.14 total 1190.09 last 72.28) - open/close 38481 (time 1786802173.14 total 1200.10 last 78.03) - open/close 39343 (time 1786802183.15 total 1210.11 last 86.12) - open/close 40000 (time 1786802191.24 total 1218.20 last 81.23) - open/close 40789 (time 1786802201.24 total 1228.20 last 78.88) - open/close 41787 (time 1786802211.26 total 1238.22 last 99.61) - open/close 42613 (time 1786802221.28 total 1248.24 last 82.43) - open/close 43220 (time 1786802231.30 total 1258.25 last 60.63) - open/close 43886 (time 1786802241.32 total 1268.28 last 66.46) - open/close 44651 (time 1786802251.32 total 1278.28 last 76.49) - open/close 45317 (time 1786802261.32 total 1288.28 last 66.59) - open/close 46079 (time 1786802271.33 total 1298.29 last 76.09) - open/close 46741 (time 1786802281.34 total 1308.30 last 66.17) - open/close 47329 (time 1786802291.35 total 1318.31 last 58.73) - open/close 47804 (time 1786802301.36 total 1328.32 last 47.45) - open/close 48528 (time 1786802311.37 total 1338.33 last 72.32) - open/close 49286 (time 1786802321.37 total 1348.33 last 75.80) total: 50000 open/close in 1357.16 seconds: 36.84 ops/second - unlinked 0 (time 1786802334 ; total 0 ; last 0) - unlinked 10000 (time 1786802482 ; total 148 ; last 148) - unlinked 20000 (time 1786802620 ; total 286 ; last 138) - unlinked 30000 (time 1786802761 ; total 427 ; last 141) - unlinked 40000 (time 1786802901 ; total 567 ; last 140) total: 50000 unlinks in 709 seconds: 70.521858 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2