== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:58:41 (1787396321) start mds service on oleg609-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg609-server' oleg609-server: oleg609-server.virtnet: executing load_modules_local oleg609-server: Loading modules from /home/green/git/lustre-release/lustre oleg609-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg609-server: detected 4 online CPUs by sysfs oleg609-server: MODOPTS_LIBCFS= oleg609-server: Force libcfs to create 2 CPU partitions oleg609-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg609-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg609-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg609-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg609-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Started lustre-OST0000 oleg609-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg609-client.virtnet: -o user_xattr,flock 192.168.206.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 252 (time 1787396401.18 total 10.26 last 24.57) - open/close 509 (time 1787396411.68 total 20.75 last 24.49) - open/close 729 (time 1787396421.84 total 30.92 last 21.64) - open/close 999 (time 1787396431.86 total 40.94 last 26.95) - open/close 1178 (time 1787396442.59 total 51.67 last 16.68) - open/close 1458 (time 1787396452.90 total 61.98 last 27.15) - open/close 1611 (time 1787396462.92 total 71.99 last 15.28) - open/close 1738 (time 1787396472.94 total 82.02 last 12.67) - open/close 1893 (time 1787396483.60 total 92.68 last 14.55) - open/close 2065 (time 1787396494.23 total 103.31 last 16.18) - open/close 2166 (time 1787396504.97 total 114.05 last 9.40) - open/close 2397 (time 1787396514.98 total 124.06 last 23.08) - open/close 2611 (time 1787396525.82 total 134.89 last 19.74) - open/close 2687 (time 1787396535.82 total 144.90 last 7.60) - open/close 2811 (time 1787396545.83 total 154.90 last 12.40) - open/close 3067 (time 1787396555.83 total 164.91 last 25.59) - open/close 3295 (time 1787396565.97 total 175.05 last 22.48) - open/close 3501 (time 1787396576.60 total 185.67 last 19.39) - open/close 3721 (time 1787396586.61 total 195.68 last 21.98) - open/close 3992 (time 1787396597.59 total 206.66 last 24.68) - open/close 4269 (time 1787396608.54 total 217.62 last 25.28) - open/close 4529 (time 1787396619.62 total 228.69 last 23.48) - open/close 4716 (time 1787396629.65 total 238.73 last 18.63) - open/close 4941 (time 1787396639.70 total 248.78 last 22.39) - open/close 5195 (time 1787396649.72 total 258.79 last 25.36) - open/close 5275 (time 1787396659.72 total 268.79 last 8.00) - open/close 5554 (time 1787396669.74 total 278.81 last 27.85) - open/close 5729 (time 1787396679.74 total 288.82 last 17.49) - open/close 5999 (time 1787396689.77 total 298.85 last 26.92) - open/close 6227 (time 1787396699.94 total 309.02 last 22.42) - open/close 6508 (time 1787396710.82 total 319.90 last 25.83) - open/close 6752 (time 1787396721.04 total 330.12 last 23.88) - open/close 7006 (time 1787396731.04 total 340.12 last 25.39) - open/close 7269 (time 1787396741.06 total 350.14 last 26.25) - open/close 7549 (time 1787396752.05 total 361.12 last 25.49) - open/close 7806 (time 1787396762.46 total 371.53 last 24.69) - open/close 8045 (time 1787396772.47 total 381.55 last 23.86) - open/close 8205 (time 1787396783.23 total 392.31 last 14.87) - open/close 8258 (time 1787396793.94 total 403.02 last 4.95) - open/close 8453 (time 1787396803.98 total 413.06 last 19.41) - open/close 8623 (time 1787396815.46 total 424.53 last 14.82) - open/close 8917 (time 1787396826.25 total 435.33 last 27.24) - open/close 9187 (time 1787396836.50 total 445.57 last 26.35) - open/close 9392 (time 1787396847.07 total 456.15 last 19.39) - open/close 9629 (time 1787396857.61 total 466.68 last 22.50) - open/close 9885 (time 1787396868.51 total 477.59 last 23.48) - open/close 10000 (time 1787396875.55 total 484.63 last 16.33) - open/close 10270 (time 1787396887.84 total 496.91 last 21.98) - open/close 10468 (time 1787396897.84 total 506.92 last 19.79) - open/close 10700 (time 1787396908.08 total 517.15 last 22.67) - open/close 10961 (time 1787396919.05 total 528.13 last 23.78) - open/close 11173 (time 1787396929.05 total 538.13 last 21.20) - open/close 11488 (time 1787396939.67 total 548.74 last 29.68) - open/close 11707 (time 1787396949.68 total 558.76 last 21.86) - open/close 11935 (time 1787396959.71 total 568.78 last 22.74) - open/close 12211 (time 1787396969.88 total 578.96 last 27.12) - open/close 12477 (time 1787396979.88 total 588.96 last 26.60) - open/close 12732 (time 1787396990.62 total 599.69 last 23.76) - open/close 13001 (time 1787397000.99 total 610.06 last 25.93) - open/close 13223 (time 1787397011.23 total 620.30 last 21.68) - open/close 13474 (time 1787397021.65 total 630.72 last 24.10) - open/close 13720 (time 1787397031.66 total 640.73 last 24.57) - open/close 13974 (time 1787397041.82 total 650.90 last 24.99) - open/close 14206 (time 1787397052.70 total 661.78 last 21.32) - open/close 14494 (time 1787397062.71 total 671.79 last 28.78) - open/close 14737 (time 1787397072.99 total 682.06 last 23.65) - open/close 14882 (time 1787397083.29 total 692.37 last 14.07) - open/close 15010 (time 1787397093.31 total 702.39 last 12.77) - open/close 15190 (time 1787397104.08 total 713.16 last 16.71) - open/close 15410 (time 1787397114.08 total 723.16 last 22.00) - open/close 15657 (time 1787397124.09 total 733.17 last 24.68) - open/close 15858 (time 1787397134.11 total 743.19 last 20.06) - open/close 15988 (time 1787397144.79 total 753.87 last 12.17) - open/close 16069 (time 1787397154.82 total 763.90 last 8.08) - open/close 16239 (time 1787397164.83 total 773.90 last 16.99) - open/close 16380 (time 1787397174.90 total 783.98 last 14.00) - open/close 16569 (time 1787397184.99 total 794.07 last 18.73) - open/close 16748 (time 1787397195.79 total 804.87 last 16.58) - open/close 16963 (time 1787397205.90 total 814.98 last 21.26) - open/close 17249 (time 1787397215.90 total 824.98 last 28.60) - open/close 17466 (time 1787397226.21 total 835.29 last 21.05) - open/close 17701 (time 1787397236.23 total 845.30 last 23.47) - open/close 17908 (time 1787397246.25 total 855.33 last 20.65) - open/close 18044 (time 1787397257.18 total 866.25 last 12.45) - open/close 18100 (time 1787397267.80 total 876.88 last 5.27) - open/close 18374 (time 1787397278.44 total 887.51 last 25.77) - open/close 18543 (time 1787397288.74 total 897.81 last 16.41) - open/close 18772 (time 1787397299.41 total 908.49 last 21.45) - open/close 18914 (time 1787397309.42 total 918.49 last 14.19) - open/close 19203 (time 1787397321.82 total 930.90 last 23.29) - open/close 19378 (time 1787397333.33 total 942.41 last 15.20) - open/close 19940 (time 1787397343.35 total 952.43 last 56.11) - open/close 20768 (time 1787397353.36 total 962.44 last 82.69) - open/close 21614 (time 1787397363.38 total 972.46 last 84.43) - open/close 22172 (time 1787397373.39 total 982.47 last 55.75) - open/close 22929 (time 1787397383.39 total 992.47 last 75.69) - open/close 23790 (time 1787397393.40 total 1002.47 last 86.07) - open/close 24589 (time 1787397403.40 total 1012.47 last 79.89) - open/close 25343 (time 1787397413.40 total 1022.48 last 75.36) - open/close 26217 (time 1787397423.41 total 1032.48 last 87.37) - open/close 27025 (time 1787397433.41 total 1042.48 last 80.80) - open/close 27833 (time 1787397443.41 total 1052.49 last 80.76) - open/close 28654 (time 1787397453.42 total 1062.49 last 82.07) - open/close 29403 (time 1787397463.45 total 1072.53 last 74.65) - open/close 30000 (time 1787397471.10 total 1080.18 last 78.03) - open/close 30802 (time 1787397481.11 total 1090.18 last 80.15) - open/close 31628 (time 1787397491.11 total 1100.19 last 82.57) - open/close 32470 (time 1787397501.12 total 1110.19 last 84.15) - open/close 33319 (time 1787397511.13 total 1120.20 last 84.83) - open/close 34135 (time 1787397521.13 total 1130.21 last 81.57) - open/close 34984 (time 1787397531.13 total 1140.21 last 84.86) - open/close 35787 (time 1787397541.14 total 1150.22 last 80.23) - open/close 36622 (time 1787397551.14 total 1160.22 last 83.49) - open/close 37467 (time 1787397561.15 total 1170.22 last 84.46) - open/close 38269 (time 1787397571.15 total 1180.23 last 80.20) - open/close 39104 (time 1787397581.16 total 1190.23 last 83.44) - open/close 39766 (time 1787397591.16 total 1200.24 last 66.15) - open/close 40000 (time 1787397594.74 total 1203.82 last 65.43) - open/close 40805 (time 1787397604.74 total 1213.82 last 80.49) - open/close 41681 (time 1787397614.77 total 1223.85 last 87.32) - open/close 42489 (time 1787397624.77 total 1233.85 last 80.80) - open/close 43288 (time 1787397634.78 total 1243.86 last 79.81) - open/close 44161 (time 1787397644.79 total 1253.87 last 87.24) - open/close 44929 (time 1787397654.79 total 1263.87 last 76.79) - open/close 45722 (time 1787397664.80 total 1273.87 last 79.27) - open/close 46533 (time 1787397674.80 total 1283.88 last 81.05) - open/close 47355 (time 1787397684.80 total 1293.88 last 82.20) - open/close 48185 (time 1787397694.81 total 1303.88 last 82.98) - open/close 49032 (time 1787397704.83 total 1313.90 last 84.52) - open/close 49787 (time 1787397714.83 total 1323.91 last 75.46) total: 50000 open/close in 1326.45 seconds: 37.69 ops/second - unlinked 0 (time 1787397720 ; total 0 ; last 0) - unlinked 10000 (time 1787397853 ; total 133 ; last 133) - unlinked 20000 (time 1787397985 ; total 265 ; last 132) - unlinked 30000 (time 1787398122 ; total 402 ; last 137) - unlinked 40000 (time 1787398255 ; total 535 ; last 133) total: 50000 unlinks in 672 seconds: 74.404762 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg609-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg609-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg609-server stop mds service on oleg609-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg609-server unloading modules via unload_modules_local on: 'oleg609-server' oleg609-server: oleg609-server.virtnet: executing unload_modules_local oleg609-server: modules unloaded. pdsh@oleg609-client: oleg609-client: ssh exited with exit code 2 pdsh@oleg609-client: oleg609-server: ssh exited with exit code 2 pdsh@oleg609-client: oleg609-client: ssh exited with exit code 2 pdsh@oleg609-client: oleg609-server: ssh exited with exit code 2