== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:41:58 (1786567318) start mds service on oleg320-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions oleg320-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg320-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg320-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1786567397.70 total 10.35 last 22.32) - open/close 481 (time 1786567407.72 total 20.37 last 24.96) - open/close 762 (time 1786567418.45 total 31.09 last 26.19) - open/close 958 (time 1786567428.45 total 41.09 last 19.60) - open/close 1212 (time 1786567438.46 total 51.10 last 25.39) - open/close 1444 (time 1786567448.91 total 61.55 last 22.20) - open/close 1697 (time 1786567459.98 total 72.62 last 22.86) - open/close 1924 (time 1786567470.95 total 83.59 last 20.69) - open/close 2132 (time 1786567481.48 total 94.12 last 19.75) - open/close 2319 (time 1786567491.55 total 104.19 last 18.57) - open/close 2491 (time 1786567501.66 total 114.31 last 17.01) - open/close 2738 (time 1786567511.68 total 124.32 last 24.65) - open/close 2940 (time 1786567521.69 total 134.34 last 20.17) - open/close 3142 (time 1786567531.72 total 144.36 last 20.15) - open/close 3341 (time 1786567541.73 total 154.37 last 19.89) - open/close 3563 (time 1786567551.74 total 164.38 last 22.18) - open/close 3816 (time 1786567562.76 total 175.41 last 22.94) - open/close 3891 (time 1786567572.77 total 185.41 last 7.50) - open/close 4094 (time 1786567583.18 total 195.83 last 19.49) - open/close 4336 (time 1786567593.20 total 205.84 last 24.17) - open/close 4545 (time 1786567604.00 total 216.64 last 19.34) - open/close 4723 (time 1786567614.22 total 226.87 last 17.41) - open/close 4858 (time 1786567624.24 total 236.88 last 13.48) - open/close 5060 (time 1786567634.63 total 247.28 last 19.43) - open/close 5239 (time 1786567644.65 total 257.30 last 17.86) - open/close 5443 (time 1786567654.66 total 267.30 last 20.39) - open/close 5548 (time 1786567664.76 total 277.41 last 10.39) - open/close 5629 (time 1786567677.51 total 290.15 last 6.36) - open/close 5786 (time 1786567688.01 total 300.66 last 14.94) - open/close 5945 (time 1786567698.31 total 310.96 last 15.44) - open/close 6146 (time 1786567708.34 total 320.98 last 20.05) - open/close 6372 (time 1786567718.61 total 331.25 last 22.00) - open/close 6620 (time 1786567728.96 total 341.60 last 23.96) - open/close 6837 (time 1786567739.20 total 351.84 last 21.20) - open/close 7072 (time 1786567749.33 total 361.97 last 23.20) - open/close 7287 (time 1786567759.58 total 372.23 last 20.96) - open/close 7510 (time 1786567770.01 total 382.65 last 21.40) - open/close 7763 (time 1786567780.35 total 393.00 last 24.45) - open/close 8034 (time 1786567790.37 total 403.02 last 27.04) - open/close 8249 (time 1786567800.46 total 413.11 last 21.31) - open/close 8485 (time 1786567811.21 total 423.85 last 21.97) - open/close 8739 (time 1786567821.21 total 433.85 last 25.40) - open/close 8948 (time 1786567831.28 total 443.92 last 20.76) - open/close 9177 (time 1786567841.89 total 454.53 last 21.58) - open/close 9306 (time 1786567851.91 total 464.55 last 12.88) - open/close 9439 (time 1786567861.91 total 474.55 last 13.30) - open/close 9554 (time 1786567871.91 total 484.56 last 11.50) - open/close 9655 (time 1786567885.58 total 498.23 last 7.39) - open/close 9863 (time 1786567896.07 total 508.71 last 19.83) - open/close 10000 (time 1786567904.11 total 516.76 last 17.04) - open/close 10187 (time 1786567915.11 total 527.75 last 17.00) - open/close 10442 (time 1786567926.15 total 538.79 last 23.10) - open/close 10522 (time 1786567938.29 total 550.93 last 6.59) - open/close 10672 (time 1786567950.32 total 562.96 last 12.47) - open/close 10796 (time 1786567960.32 total 572.97 last 12.39) - open/close 10942 (time 1786567970.93 total 583.57 last 13.77) - open/close 11131 (time 1786567980.94 total 593.58 last 18.88) - open/close 11225 (time 1786567991.20 total 603.84 last 9.16) - open/close 11434 (time 1786568002.15 total 614.79 last 19.09) - open/close 11624 (time 1786568013.24 total 625.88 last 17.13) - open/close 11733 (time 1786568023.88 total 636.52 last 10.24) - open/close 11982 (time 1786568033.90 total 646.54 last 24.86) - open/close 12200 (time 1786568049.94 total 662.58 last 13.59) - open/close 12364 (time 1786568060.17 total 672.81 last 16.03) - open/close 12470 (time 1786568070.18 total 682.82 last 10.59) - open/close 12652 (time 1786568080.44 total 693.09 last 17.74) - open/close 12876 (time 1786568091.20 total 703.84 last 20.82) - open/close 13074 (time 1786568101.65 total 714.29 last 18.95) - open/close 13188 (time 1786568111.67 total 724.31 last 11.37) - open/close 13322 (time 1786568121.74 total 734.39 last 13.30) - open/close 13523 (time 1786568134.28 total 746.92 last 16.03) - open/close 13723 (time 1786568144.29 total 756.94 last 19.97) - open/close 13945 (time 1786568154.29 total 766.94 last 22.19) - open/close 14147 (time 1786568164.56 total 777.21 last 19.67) - open/close 14359 (time 1786568175.19 total 787.83 last 19.95) - open/close 14618 (time 1786568185.82 total 798.47 last 24.36) - open/close 14865 (time 1786568195.83 total 808.48 last 24.68) - open/close 15048 (time 1786568205.84 total 818.49 last 18.28) - open/close 15286 (time 1786568216.66 total 829.30 last 22.00) - open/close 15514 (time 1786568226.66 total 839.31 last 22.79) - open/close 15760 (time 1786568236.68 total 849.33 last 24.55) - open/close 16047 (time 1786568246.69 total 859.33 last 28.69) - open/close 16297 (time 1786568256.70 total 869.34 last 24.97) - open/close 16540 (time 1786568266.70 total 879.34 last 24.30) - open/close 16771 (time 1786568276.73 total 889.37 last 23.04) - open/close 16951 (time 1786568286.75 total 899.39 last 17.96) - open/close 17155 (time 1786568297.22 total 909.87 last 19.48) - open/close 17423 (time 1786568308.10 total 920.75 last 24.63) - open/close 17673 (time 1786568318.11 total 930.75 last 24.99) - open/close 17941 (time 1786568328.11 total 940.75 last 26.79) - open/close 18159 (time 1786568338.13 total 950.77 last 21.76) - open/close 18643 (time 1786568348.14 total 960.78 last 48.34) - open/close 19284 (time 1786568358.15 total 970.79 last 64.06) - open/close 19951 (time 1786568368.15 total 980.80 last 66.65) - open/close 20643 (time 1786568378.15 total 990.80 last 69.20) - open/close 21326 (time 1786568388.16 total 1000.80 last 68.25) - open/close 22028 (time 1786568398.18 total 1010.82 last 70.10) - open/close 22699 (time 1786568408.19 total 1020.84 last 66.99) - open/close 23353 (time 1786568418.21 total 1030.86 last 65.26) - open/close 23978 (time 1786568428.23 total 1040.88 last 62.37) - open/close 24589 (time 1786568438.24 total 1050.88 last 61.09) - open/close 25270 (time 1786568448.25 total 1060.89 last 68.03) - open/close 25947 (time 1786568458.25 total 1070.90 last 67.66) - open/close 26544 (time 1786568468.26 total 1080.91 last 59.64) - open/close 27207 (time 1786568478.27 total 1090.91 last 66.28) - open/close 27895 (time 1786568488.27 total 1100.92 last 68.76) - open/close 28587 (time 1786568498.28 total 1110.92 last 69.18) - open/close 29259 (time 1786568508.29 total 1120.93 last 67.13) - open/close 29918 (time 1786568518.29 total 1130.93 last 65.88) - open/close 30585 (time 1786568528.30 total 1140.94 last 66.64) - open/close 31274 (time 1786568538.31 total 1150.95 last 68.81) - open/close 31976 (time 1786568548.32 total 1160.96 last 70.14) - open/close 32673 (time 1786568558.33 total 1170.97 last 69.61) - open/close 33343 (time 1786568568.34 total 1180.98 last 66.93) - open/close 34005 (time 1786568578.35 total 1191.00 last 66.13) - open/close 34671 (time 1786568588.38 total 1201.02 last 66.43) - open/close 35358 (time 1786568598.38 total 1211.02 last 68.70) - open/close 36051 (time 1786568608.38 total 1221.03 last 69.27) - open/close 36718 (time 1786568618.40 total 1231.04 last 66.61) - open/close 37379 (time 1786568628.40 total 1241.04 last 66.07) - open/close 38080 (time 1786568638.41 total 1251.05 last 70.04) - open/close 38713 (time 1786568648.41 total 1261.06 last 63.27) - open/close 39320 (time 1786568658.42 total 1271.06 last 60.66) - open/close 40000 (time 1786568668.37 total 1281.02 last 68.32) - open/close 40690 (time 1786568678.38 total 1291.02 last 68.96) - open/close 41432 (time 1786568688.38 total 1301.02 last 74.17) - open/close 42144 (time 1786568698.39 total 1311.03 last 71.14) - open/close 42846 (time 1786568708.40 total 1321.04 last 70.13) - open/close 43522 (time 1786568718.41 total 1331.05 last 67.55) - open/close 44214 (time 1786568728.42 total 1341.06 last 69.13) - open/close 44948 (time 1786568738.42 total 1351.06 last 73.38) - open/close 45606 (time 1786568748.43 total 1361.08 last 65.70) - open/close 46281 (time 1786568758.44 total 1371.09 last 67.44) - open/close 46955 (time 1786568768.45 total 1381.09 last 67.35) - open/close 47679 (time 1786568778.45 total 1391.10 last 72.38) - open/close 48383 (time 1786568788.46 total 1401.10 last 70.34) - open/close 49042 (time 1786568798.47 total 1411.11 last 65.87) - open/close 49726 (time 1786568808.47 total 1421.11 last 68.37) total: 50000 open/close in 1424.74 seconds: 35.09 ops/second - unlinked 0 (time 1786568816 ; total 0 ; last 0) - unlinked 10000 (time 1786568966 ; total 150 ; last 150) - unlinked 20000 (time 1786569110 ; total 294 ; last 144) - unlinked 30000 (time 1786569247 ; total 431 ; last 137) - unlinked 40000 (time 1786569386 ; total 570 ; last 139) total: 50000 unlinks in 715 seconds: 69.930069 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: modules unloaded. pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2