== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:51 (1787370471) start mds service on oleg623-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing load_modules_local oleg623-server: Loading modules from /home/green/git/lustre-release/lustre oleg623-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg623-server: detected 4 online CPUs by sysfs oleg623-server: MODOPTS_LIBCFS= oleg623-server: Force libcfs to create 2 CPU partitions oleg623-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg623-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg623-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg623-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1787370556.64 total 10.13 last 20.54) - open/close 416 (time 1787370566.65 total 20.13 last 20.79) - open/close 616 (time 1787370576.81 total 30.29 last 19.68) - open/close 892 (time 1787370586.83 total 40.31 last 27.55) - open/close 1115 (time 1787370596.84 total 50.32 last 22.28) - open/close 1214 (time 1787370608.29 total 61.77 last 8.64) - open/close 1395 (time 1787370618.32 total 71.80 last 18.06) - open/close 1690 (time 1787370628.32 total 81.81 last 29.48) - open/close 1887 (time 1787370639.08 total 92.56 last 18.31) - open/close 2095 (time 1787370649.09 total 102.57 last 20.78) - open/close 2226 (time 1787370659.11 total 112.59 last 13.08) - open/close 2458 (time 1787370669.63 total 123.11 last 22.06) - open/close 2728 (time 1787370680.36 total 133.84 last 25.16) - open/close 2924 (time 1787370690.37 total 143.86 last 19.56) - open/close 3144 (time 1787370701.26 total 154.74 last 20.22) - open/close 3311 (time 1787370711.28 total 164.76 last 16.67) - open/close 3467 (time 1787370721.29 total 174.77 last 15.58) - open/close 3587 (time 1787370731.50 total 184.98 last 11.75) - open/close 3828 (time 1787370741.50 total 194.99 last 24.10) - open/close 4022 (time 1787370751.67 total 205.15 last 19.08) - open/close 4201 (time 1787370762.53 total 216.01 last 16.48) - open/close 4389 (time 1787370772.53 total 226.02 last 18.79) - open/close 4664 (time 1787370782.54 total 236.02 last 27.48) - open/close 4909 (time 1787370792.56 total 246.05 last 24.44) - open/close 5157 (time 1787370803.46 total 256.94 last 22.77) - open/close 5393 (time 1787370813.48 total 266.96 last 23.55) - open/close 5608 (time 1787370823.78 total 277.26 last 20.87) - open/close 5867 (time 1787370834.29 total 287.77 last 24.64) - open/close 6055 (time 1787370844.39 total 297.87 last 18.61) - open/close 6258 (time 1787370854.39 total 307.88 last 20.30) - open/close 6458 (time 1787370864.82 total 318.30 last 19.19) - open/close 6705 (time 1787370874.82 total 328.31 last 24.68) - open/close 6911 (time 1787370887.09 total 340.57 last 16.79) - open/close 7111 (time 1787370898.07 total 351.55 last 18.21) - open/close 7376 (time 1787370908.09 total 361.57 last 26.45) - open/close 7546 (time 1787370918.17 total 371.66 last 16.86) - open/close 7662 (time 1787370928.18 total 381.67 last 11.59) - open/close 7802 (time 1787370938.53 total 392.01 last 13.54) - open/close 8009 (time 1787370948.90 total 402.39 last 19.95) - open/close 8246 (time 1787370959.59 total 413.07 last 22.18) - open/close 8477 (time 1787370969.96 total 423.44 last 22.27) - open/close 8713 (time 1787370980.70 total 434.18 last 21.98) - open/close 8966 (time 1787370990.74 total 444.22 last 25.21) - open/close 9108 (time 1787371001.71 total 455.19 last 12.94) - open/close 9212 (time 1787371012.41 total 465.90 last 9.71) - open/close 9454 (time 1787371022.42 total 475.90 last 24.19) - open/close 9549 (time 1787371032.55 total 486.03 last 9.38) - open/close 9755 (time 1787371043.74 total 497.23 last 18.41) - open/close 9971 (time 1787371053.79 total 507.28 last 21.49) - open/close 10209 (time 1787371064.34 total 517.82 last 22.56) - open/close 10415 (time 1787371074.71 total 528.19 last 19.87) - open/close 10451 (time 1787371085.87 total 539.35 last 3.23) - open/close 10527 (time 1787371095.88 total 549.36 last 7.59) - open/close 10730 (time 1787371105.89 total 559.37 last 20.27) - open/close 10980 (time 1787371116.15 total 569.63 last 24.37) - open/close 11138 (time 1787371126.62 total 580.10 last 15.09) - open/close 11331 (time 1787371136.62 total 590.10 last 19.30) - open/close 11453 (time 1787371146.63 total 600.11 last 12.19) - open/close 11621 (time 1787371157.42 total 610.90 last 15.56) - open/close 11861 (time 1787371167.42 total 620.90 last 24.00) - open/close 11991 (time 1787371177.51 total 630.99 last 12.89) - open/close 12236 (time 1787371187.94 total 641.42 last 23.48) - open/close 12390 (time 1787371199.27 total 652.75 last 13.59) - open/close 12653 (time 1787371209.32 total 662.80 last 26.17) - open/close 12895 (time 1787371219.33 total 672.81 last 24.17) - open/close 13031 (time 1787371231.01 total 684.50 last 11.64) - open/close 13241 (time 1787371241.67 total 695.15 last 19.71) - open/close 13455 (time 1787371252.28 total 705.76 last 20.17) - open/close 13646 (time 1787371262.55 total 716.03 last 18.59) - open/close 13919 (time 1787371272.55 total 726.04 last 27.29) - open/close 14148 (time 1787371282.91 total 736.39 last 22.11) - open/close 14341 (time 1787371292.92 total 746.40 last 19.28) - open/close 14472 (time 1787371303.07 total 756.55 last 12.91) - open/close 14706 (time 1787371313.13 total 766.61 last 23.28) - open/close 14970 (time 1787371323.13 total 776.61 last 26.38) - open/close 15183 (time 1787371333.13 total 786.61 last 21.30) - open/close 15482 (time 1787371343.86 total 797.34 last 27.88) - open/close 15750 (time 1787371353.90 total 807.38 last 26.69) - open/close 16023 (time 1787371363.92 total 817.40 last 27.24) - open/close 16180 (time 1787371374.90 total 828.39 last 14.29) - open/close 16406 (time 1787371387.63 total 841.11 last 17.77) - open/close 16658 (time 1787371397.85 total 851.34 last 24.64) - open/close 16877 (time 1787371408.48 total 861.96 last 20.61) - open/close 17144 (time 1787371419.44 total 872.92 last 24.36) - open/close 17400 (time 1787371430.25 total 883.73 last 23.68) - open/close 17641 (time 1787371440.88 total 894.36 last 22.68) - open/close 17881 (time 1787371451.37 total 904.85 last 22.87) - open/close 18131 (time 1787371462.11 total 915.59 last 23.29) - open/close 18363 (time 1787371472.43 total 925.91 last 22.49) - open/close 18597 (time 1787371482.92 total 936.40 last 22.30) - open/close 18792 (time 1787371493.36 total 946.84 last 18.67) - open/close 18984 (time 1787371504.41 total 957.89 last 17.39) - open/close 19260 (time 1787371514.42 total 967.91 last 27.55) - open/close 19916 (time 1787371524.43 total 977.91 last 65.59) - open/close 20569 (time 1787371534.43 total 987.91 last 65.28) - open/close 21302 (time 1787371544.44 total 997.92 last 73.22) - open/close 22022 (time 1787371554.44 total 1007.93 last 71.98) - open/close 22694 (time 1787371564.46 total 1017.94 last 67.08) - open/close 23368 (time 1787371574.47 total 1027.95 last 67.34) - open/close 24022 (time 1787371584.48 total 1037.96 last 65.35) - open/close 24681 (time 1787371594.51 total 1047.99 last 65.68) - open/close 25321 (time 1787371604.53 total 1058.02 last 63.86) - open/close 25922 (time 1787371614.55 total 1068.03 last 60.02) - open/close 26442 (time 1787371624.55 total 1078.03 last 51.97) - open/close 27062 (time 1787371634.57 total 1088.05 last 61.89) - open/close 27674 (time 1787371644.57 total 1098.05 last 61.19) - open/close 28233 (time 1787371654.58 total 1108.06 last 55.87) - open/close 28917 (time 1787371664.59 total 1118.07 last 68.34) - open/close 29564 (time 1787371674.59 total 1128.07 last 64.68) - open/close 30000 (time 1787371680.55 total 1134.03 last 73.19) - open/close 30762 (time 1787371690.56 total 1144.05 last 76.07) - open/close 31460 (time 1787371700.56 total 1154.05 last 69.80) - open/close 32159 (time 1787371710.57 total 1164.05 last 69.87) - open/close 32843 (time 1787371720.57 total 1174.05 last 68.38) - open/close 33515 (time 1787371730.59 total 1184.08 last 67.05) - open/close 34213 (time 1787371740.60 total 1194.08 last 69.80) - open/close 34902 (time 1787371750.61 total 1204.09 last 68.81) - open/close 35578 (time 1787371760.62 total 1214.10 last 67.54) - open/close 36266 (time 1787371770.63 total 1224.11 last 68.75) - open/close 36921 (time 1787371780.63 total 1234.11 last 65.47) - open/close 37619 (time 1787371790.64 total 1244.12 last 69.75) - open/close 38319 (time 1787371800.64 total 1254.12 last 69.97) - open/close 38973 (time 1787371810.65 total 1264.13 last 65.35) - open/close 39628 (time 1787371820.67 total 1274.15 last 65.39) - open/close 40000 (time 1787371826.11 total 1279.60 last 68.26) - open/close 40681 (time 1787371836.15 total 1289.64 last 67.84) - open/close 41363 (time 1787371846.17 total 1299.65 last 68.09) - open/close 42054 (time 1787371856.17 total 1309.66 last 69.06) - open/close 42735 (time 1787371866.19 total 1319.67 last 68.02) - open/close 43400 (time 1787371876.19 total 1329.68 last 66.45) - open/close 44111 (time 1787371886.20 total 1339.68 last 71.06) - open/close 44819 (time 1787371896.20 total 1349.68 last 70.78) - open/close 45537 (time 1787371906.22 total 1359.70 last 71.69) - open/close 46217 (time 1787371916.22 total 1369.70 last 67.99) - open/close 46869 (time 1787371926.22 total 1379.70 last 65.18) - open/close 47605 (time 1787371936.23 total 1389.71 last 73.55) - open/close 48230 (time 1787371946.24 total 1399.72 last 62.43) - open/close 48875 (time 1787371956.25 total 1409.73 last 64.43) - open/close 49462 (time 1787371966.27 total 1419.75 last 58.60) total: 50000 open/close in 1426.94 seconds: 35.04 ops/second - unlinked 0 (time 1787371978 ; total 0 ; last 0) - unlinked 10000 (time 1787372128 ; total 150 ; last 150) - unlinked 20000 (time 1787372284 ; total 306 ; last 156) - unlinked 30000 (time 1787372440 ; total 462 ; last 156) - unlinked 40000 (time 1787372591 ; total 613 ; last 151) total: 50000 unlinks in 785 seconds: 63.694267 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server stop mds service on oleg623-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg623-server unloading modules via unload_modules_local on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing unload_modules_local oleg623-server: modules unloaded. pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2