== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:48:04 (1763351284) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 230 (time 1763351357.59 total 10.47 last 21.96) - open/close 472 (time 1763351367.94 total 20.82 last 23.40) - open/close 639 (time 1763351377.94 total 30.82 last 16.69) - open/close 876 (time 1763351387.95 total 40.84 last 23.67) - open/close 1086 (time 1763351397.96 total 50.84 last 21.00) - open/close 1293 (time 1763351407.97 total 60.85 last 20.67) - open/close 1523 (time 1763351418.89 total 71.77 last 21.07) - open/close 1749 (time 1763351429.27 total 82.15 last 21.77) - open/close 1967 (time 1763351439.28 total 92.16 last 21.77) - open/close 2152 (time 1763351450.13 total 103.01 last 17.06) - open/close 2389 (time 1763351460.82 total 113.70 last 22.16) - open/close 2623 (time 1763351470.85 total 123.73 last 23.33) - open/close 2851 (time 1763351481.42 total 134.30 last 21.58) - open/close 3068 (time 1763351491.43 total 144.31 last 21.67) - open/close 3292 (time 1763351502.09 total 154.98 last 21.01) - open/close 3493 (time 1763351512.11 total 164.99 last 20.07) - open/close 3706 (time 1763351522.33 total 175.21 last 20.84) - open/close 3956 (time 1763351532.34 total 185.22 last 24.98) - open/close 4182 (time 1763351542.73 total 195.61 last 21.76) - open/close 4406 (time 1763351553.10 total 205.98 last 21.60) - open/close 4601 (time 1763351563.85 total 216.73 last 18.13) - open/close 4807 (time 1763351573.87 total 226.76 last 20.56) - open/close 5015 (time 1763351583.88 total 236.76 last 20.79) - open/close 5268 (time 1763351594.90 total 247.78 last 22.96) - open/close 5477 (time 1763351604.92 total 257.80 last 20.86) - open/close 5692 (time 1763351614.98 total 267.86 last 21.37) - open/close 5916 (time 1763351624.98 total 277.86 last 22.40) - open/close 6136 (time 1763351635.48 total 288.36 last 20.96) - open/close 6358 (time 1763351645.49 total 298.37 last 22.16) - open/close 6564 (time 1763351655.56 total 308.44 last 20.47) - open/close 6770 (time 1763351665.56 total 318.44 last 20.60) - open/close 6981 (time 1763351675.73 total 328.61 last 20.74) - open/close 7204 (time 1763351685.75 total 338.63 last 22.25) - open/close 7415 (time 1763351695.77 total 348.65 last 21.05) - open/close 7624 (time 1763351706.57 total 359.45 last 19.36) - open/close 7860 (time 1763351717.01 total 369.89 last 22.60) - open/close 8039 (time 1763351727.31 total 380.19 last 17.38) - open/close 8220 (time 1763351738.20 total 391.08 last 16.62) - open/close 8426 (time 1763351749.00 total 401.88 last 19.08) - open/close 8589 (time 1763351759.01 total 411.89 last 16.29) - open/close 8792 (time 1763351769.60 total 422.48 last 19.16) - open/close 8966 (time 1763351779.98 total 432.86 last 16.77) - open/close 9102 (time 1763351790.40 total 443.28 last 13.05) - open/close 9239 (time 1763351800.70 total 453.58 last 13.30) - open/close 9428 (time 1763351811.01 total 463.90 last 18.33) - open/close 9605 (time 1763351821.02 total 473.90 last 17.70) - open/close 9807 (time 1763351831.37 total 484.25 last 19.51) - open/close 9963 (time 1763351842.12 total 495.00 last 14.51) - open/close 10167 (time 1763351852.62 total 505.50 last 19.42) - open/close 10350 (time 1763351862.63 total 515.51 last 18.29) - open/close 10511 (time 1763351872.64 total 525.52 last 16.08) - open/close 10674 (time 1763351883.53 total 536.41 last 14.97) - open/close 10882 (time 1763351893.54 total 546.42 last 20.79) - open/close 11026 (time 1763351904.07 total 556.95 last 13.68) - open/close 11184 (time 1763351914.65 total 567.53 last 14.93) - open/close 11384 (time 1763351925.51 total 578.39 last 18.42) - open/close 11617 (time 1763351936.32 total 589.20 last 21.55) - open/close 11744 (time 1763351946.33 total 599.21 last 12.68) - open/close 11859 (time 1763351956.34 total 609.22 last 11.49) - open/close 12015 (time 1763351966.85 total 619.73 last 14.85) - open/close 12175 (time 1763351977.54 total 630.42 last 14.96) - open/close 12305 (time 1763351987.91 total 640.79 last 12.54) - open/close 12496 (time 1763351998.02 total 650.90 last 18.88) - open/close 12691 (time 1763352008.20 total 661.08 last 19.16) - open/close 12863 (time 1763352019.20 total 672.08 last 15.63) - open/close 12979 (time 1763352030.22 total 683.10 last 10.53) - open/close 13062 (time 1763352040.70 total 693.58 last 7.91) - open/close 13188 (time 1763352050.71 total 703.59 last 12.60) - open/close 13279 (time 1763352061.25 total 714.13 last 8.63) - open/close 13400 (time 1763352071.26 total 724.14 last 12.09) - open/close 13601 (time 1763352081.74 total 734.62 last 19.19) - open/close 14441 (time 1763352091.74 total 744.62 last 83.95) - open/close 15343 (time 1763352101.75 total 754.63 last 90.18) - open/close 16125 (time 1763352111.75 total 764.63 last 78.19) - open/close 17042 (time 1763352121.75 total 774.63 last 91.67) - open/close 17928 (time 1763352131.75 total 784.63 last 88.57) - open/close 18735 (time 1763352141.76 total 794.64 last 80.66) - open/close 19612 (time 1763352151.76 total 804.64 last 87.66) - open/close 20000 (time 1763352156.19 total 809.08 last 87.54) - open/close 21056 (time 1763352166.20 total 819.08 last 105.60) - open/close 21918 (time 1763352176.20 total 829.08 last 86.17) - open/close 22833 (time 1763352186.20 total 839.08 last 91.47) - open/close 23803 (time 1763352196.20 total 849.08 last 96.98) - open/close 24710 (time 1763352206.21 total 859.09 last 90.64) - open/close 25526 (time 1763352216.23 total 869.11 last 81.42) - open/close 26389 (time 1763352226.24 total 879.12 last 86.25) - open/close 27259 (time 1763352236.24 total 889.12 last 86.99) - open/close 28083 (time 1763352246.24 total 899.12 last 82.38) - open/close 28998 (time 1763352256.26 total 909.14 last 91.38) - open/close 29887 (time 1763352266.26 total 919.14 last 88.86) - open/close 30714 (time 1763352276.27 total 929.15 last 82.60) - open/close 31525 (time 1763352286.28 total 939.16 last 81.07) - open/close 32344 (time 1763352296.28 total 949.16 last 81.86) - open/close 33298 (time 1763352306.29 total 959.17 last 95.32) - open/close 34278 (time 1763352316.30 total 969.18 last 97.93) - open/close 35249 (time 1763352326.30 total 979.18 last 97.08) - open/close 36233 (time 1763352336.31 total 989.19 last 98.30) - open/close 37072 (time 1763352346.31 total 999.19 last 83.86) - open/close 37981 (time 1763352356.32 total 1009.20 last 90.84) - open/close 38828 (time 1763352366.32 total 1019.21 last 84.67) - open/close 39847 (time 1763352376.33 total 1029.21 last 101.87) - open/close 40662 (time 1763352386.34 total 1039.22 last 81.40) - open/close 41659 (time 1763352396.35 total 1049.23 last 99.63) - open/close 42595 (time 1763352406.35 total 1059.23 last 93.58) - open/close 43552 (time 1763352416.35 total 1069.23 last 95.66) - open/close 44394 (time 1763352426.36 total 1079.24 last 84.17) - open/close 45268 (time 1763352436.36 total 1089.24 last 87.38) - open/close 46075 (time 1763352446.36 total 1099.24 last 80.66) - open/close 46877 (time 1763352456.37 total 1109.25 last 80.15) - open/close 47696 (time 1763352466.38 total 1119.26 last 81.80) - open/close 48530 (time 1763352476.39 total 1129.27 last 83.32) - open/close 49443 (time 1763352486.40 total 1139.28 last 91.25) total: 50000 open/close in 1145.49 seconds: 43.65 ops/second - unlinked 0 (time 1763352495 ; total 0 ; last 0) - unlinked 10000 (time 1763352612 ; total 117 ; last 117) - unlinked 20000 (time 1763352695 ; total 200 ; last 83) - unlinked 30000 (time 1763352751 ; total 256 ; last 56) - unlinked 40000 (time 1763352811 ; total 316 ; last 60) total: 50000 unlinks in 369 seconds: 135.501358 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2