== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:14:04 (1775196844) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg334-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1775196930.34 total 11.06 last 23.32) - open/close 485 (time 1775196943.16 total 23.89 last 17.70) - open/close 698 (time 1775196953.43 total 34.15 last 20.75) - open/close 810 (time 1775196963.45 total 44.17 last 11.18) - open/close 908 (time 1775196973.73 total 54.46 last 9.53) - open/close 1108 (time 1775196984.09 total 64.81 last 19.31) - open/close 1184 (time 1775196994.14 total 74.87 last 7.56) - open/close 1402 (time 1775197004.14 total 84.87 last 21.79) - open/close 1544 (time 1775197014.37 total 95.09 last 13.89) - open/close 1745 (time 1775197024.40 total 105.12 last 20.05) - open/close 1926 (time 1775197035.35 total 116.08 last 16.52) - open/close 2110 (time 1775197045.37 total 126.09 last 18.37) - open/close 2308 (time 1775197055.66 total 136.39 last 19.23) - open/close 2381 (time 1775197067.61 total 148.33 last 6.11) - open/close 2626 (time 1775197078.18 total 158.90 last 23.18) - open/close 2849 (time 1775197089.26 total 169.99 last 20.12) - open/close 3005 (time 1775197099.55 total 180.27 last 15.17) - open/close 3120 (time 1775197110.62 total 191.35 last 10.38) - open/close 3358 (time 1775197120.86 total 201.59 last 23.25) - open/close 3610 (time 1775197131.80 total 212.52 last 23.04) - open/close 3858 (time 1775197142.29 total 223.01 last 23.65) - open/close 4068 (time 1775197153.24 total 233.96 last 19.18) - open/close 4300 (time 1775197164.12 total 244.84 last 21.31) - open/close 4556 (time 1775197175.02 total 255.74 last 23.50) - open/close 4698 (time 1775197185.03 total 265.75 last 14.18) - open/close 4926 (time 1775197196.63 total 277.36 last 19.65) - open/close 5168 (time 1775197206.65 total 287.37 last 24.16) - open/close 5367 (time 1775197216.65 total 297.38 last 19.89) - open/close 5465 (time 1775197226.97 total 307.69 last 9.50) - open/close 5589 (time 1775197236.98 total 317.70 last 12.39) - open/close 5738 (time 1775197247.00 total 327.73 last 14.86) - open/close 5940 (time 1775197258.15 total 338.87 last 18.12) - open/close 6129 (time 1775197268.15 total 348.88 last 18.89) - open/close 6385 (time 1775197278.16 total 358.88 last 25.60) - open/close 6612 (time 1775197288.18 total 368.90 last 22.65) - open/close 6818 (time 1775197301.29 total 382.01 last 15.72) - open/close 7057 (time 1775197311.31 total 392.03 last 23.84) - open/close 7276 (time 1775197321.31 total 402.04 last 21.90) - open/close 7528 (time 1775197331.33 total 412.05 last 25.16) - open/close 7773 (time 1775197342.09 total 422.82 last 22.76) - open/close 7972 (time 1775197352.29 total 433.01 last 19.52) - open/close 8170 (time 1775197362.30 total 443.02 last 19.79) - open/close 8402 (time 1775197372.67 total 453.39 last 22.37) - open/close 8636 (time 1775197382.67 total 463.40 last 23.39) - open/close 8786 (time 1775197392.70 total 473.42 last 14.96) - open/close 9012 (time 1775197402.87 total 483.59 last 22.23) - open/close 9277 (time 1775197413.65 total 494.38 last 24.57) - open/close 9516 (time 1775197423.69 total 504.42 last 23.80) - open/close 9717 (time 1775197433.69 total 514.42 last 20.10) - open/close 9906 (time 1775197444.13 total 524.85 last 18.11) - open/close 10000 (time 1775197446.65 total 527.37 last 37.29) - open/close 10222 (time 1775197456.65 total 537.38 last 22.19) - open/close 10505 (time 1775197467.48 total 548.20 last 26.14) - open/close 10727 (time 1775197478.42 total 559.14 last 20.29) - open/close 10941 (time 1775197488.43 total 569.16 last 21.37) - open/close 11197 (time 1775197498.44 total 579.17 last 25.57) - open/close 11426 (time 1775197509.46 total 590.18 last 20.79) - open/close 11656 (time 1775197519.77 total 600.49 last 22.32) - open/close 11900 (time 1775197529.77 total 610.49 last 24.39) - open/close 12123 (time 1775197542.13 total 622.86 last 18.03) - open/close 12277 (time 1775197552.81 total 633.53 last 14.43) - open/close 12477 (time 1775197563.74 total 644.46 last 18.30) - open/close 12695 (time 1775197574.10 total 654.82 last 21.05) - open/close 12910 (time 1775197584.40 total 665.13 last 20.87) - open/close 13121 (time 1775197594.53 total 675.26 last 20.83) - open/close 13345 (time 1775197604.54 total 685.26 last 22.39) - open/close 13578 (time 1775197615.09 total 695.82 last 22.08) - open/close 13837 (time 1775197625.20 total 705.92 last 25.63) - open/close 14069 (time 1775197635.61 total 716.33 last 22.28) - open/close 14354 (time 1775197645.63 total 726.36 last 28.43) - open/close 14571 (time 1775197656.33 total 737.06 last 20.28) - open/close 14788 (time 1775197666.35 total 747.07 last 21.68) - open/close 14988 (time 1775197676.49 total 757.21 last 19.71) - open/close 15156 (time 1775197686.50 total 767.22 last 16.78) - open/close 15353 (time 1775197699.51 total 780.23 last 15.15) - open/close 15483 (time 1775197711.15 total 791.88 last 11.16) - open/close 15715 (time 1775197722.21 total 802.94 last 20.98) - open/close 15957 (time 1775197732.22 total 812.95 last 24.17) - open/close 16156 (time 1775197742.31 total 823.04 last 19.72) - open/close 16316 (time 1775197752.34 total 833.07 last 15.95) - open/close 16491 (time 1775197763.25 total 843.97 last 16.05) - open/close 16656 (time 1775197773.91 total 854.64 last 15.47) - open/close 16891 (time 1775197784.41 total 865.14 last 22.38) - open/close 17069 (time 1775197795.10 total 875.83 last 16.65) - open/close 17292 (time 1775197805.16 total 885.89 last 22.17) - open/close 17519 (time 1775197815.51 total 896.23 last 21.95) - open/close 17745 (time 1775197827.83 total 908.56 last 18.34) - open/close 17966 (time 1775197839.70 total 920.42 last 18.62) - open/close 18566 (time 1775197849.71 total 930.43 last 59.95) - open/close 19219 (time 1775197859.72 total 940.44 last 65.24) - open/close 19844 (time 1775197869.73 total 950.45 last 62.44) - open/close 20000 (time 1775197872.23 total 952.96 last 62.30) - open/close 20656 (time 1775197882.24 total 962.96 last 65.58) - open/close 21297 (time 1775197892.25 total 972.97 last 64.04) - open/close 21928 (time 1775197902.25 total 982.97 last 63.07) - open/close 22648 (time 1775197912.27 total 992.99 last 71.89) - open/close 23347 (time 1775197922.28 total 1003.00 last 69.82) - open/close 24105 (time 1775197932.29 total 1013.01 last 75.70) - open/close 24881 (time 1775197942.30 total 1023.02 last 77.52) - open/close 25664 (time 1775197952.30 total 1033.03 last 78.29) - open/close 26418 (time 1775197962.31 total 1043.04 last 75.33) - open/close 27141 (time 1775197972.33 total 1053.05 last 72.18) - open/close 27831 (time 1775197982.34 total 1063.07 last 68.88) - open/close 28528 (time 1775197992.37 total 1073.09 last 69.55) - open/close 29264 (time 1775198002.37 total 1083.09 last 73.59) - open/close 29948 (time 1775198012.37 total 1093.10 last 68.36) - open/close 30705 (time 1775198022.39 total 1103.11 last 75.60) - open/close 31455 (time 1775198032.39 total 1113.12 last 74.93) - open/close 32170 (time 1775198042.40 total 1123.13 last 71.44) - open/close 32944 (time 1775198052.42 total 1133.14 last 77.29) - open/close 33636 (time 1775198062.42 total 1143.15 last 69.17) - open/close 34261 (time 1775198072.43 total 1153.15 last 62.45) - open/close 35006 (time 1775198082.43 total 1163.15 last 74.50) - open/close 35759 (time 1775198092.43 total 1173.15 last 75.30) - open/close 36485 (time 1775198102.44 total 1183.16 last 72.55) - open/close 37177 (time 1775198112.44 total 1193.17 last 69.16) - open/close 37874 (time 1775198122.45 total 1203.17 last 69.66) - open/close 38550 (time 1775198132.45 total 1213.17 last 67.60) - open/close 39277 (time 1775198142.48 total 1223.20 last 72.48) - open/close 39960 (time 1775198152.49 total 1233.22 last 68.22) - open/close 40649 (time 1775198162.49 total 1243.22 last 68.89) - open/close 41366 (time 1775198172.49 total 1253.22 last 71.68) - open/close 42100 (time 1775198182.51 total 1263.23 last 73.29) - open/close 42762 (time 1775198192.51 total 1273.24 last 66.18) - open/close 43330 (time 1775198202.53 total 1283.25 last 56.71) - open/close 43888 (time 1775198212.55 total 1293.27 last 55.71) - open/close 44410 (time 1775198222.55 total 1303.27 last 52.19) - open/close 45127 (time 1775198232.55 total 1313.28 last 71.66) - open/close 45901 (time 1775198242.56 total 1323.29 last 77.32) - open/close 46591 (time 1775198252.57 total 1333.29 last 68.99) - open/close 47302 (time 1775198262.57 total 1343.29 last 71.08) - open/close 48029 (time 1775198272.57 total 1353.30 last 72.66) - open/close 48744 (time 1775198282.60 total 1363.32 last 71.34) - open/close 49486 (time 1775198292.60 total 1373.32 last 74.19) total: 50000 open/close in 1381.04 seconds: 36.20 ops/second - unlinked 0 (time 1775198304 ; total 0 ; last 0) - unlinked 10000 (time 1775198451 ; total 147 ; last 147) - unlinked 20000 (time 1775198599 ; total 295 ; last 148) - unlinked 30000 (time 1775198758 ; total 454 ; last 159) - unlinked 40000 (time 1775198908 ; total 604 ; last 150) total: 50000 unlinks in 756 seconds: 66.137566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2