== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:53:18 (1787604798) start mds service on oleg233-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg233-server: detected 4 online CPUs by sysfs oleg233-server: MODOPTS_LIBCFS= oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg233-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg233-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 217 (time 1787604883.76 total 12.10 last 17.94) - open/close 473 (time 1787604893.76 total 22.10 last 25.59) - open/close 587 (time 1787604903.78 total 32.12 last 11.38) - open/close 678 (time 1787604913.92 total 42.26 last 8.98) - open/close 883 (time 1787604924.72 total 53.06 last 18.97) - open/close 1108 (time 1787604934.73 total 63.07 last 22.48) - open/close 1321 (time 1787604945.51 total 73.85 last 19.76) - open/close 1472 (time 1787604956.26 total 84.60 last 14.05) - open/close 1637 (time 1787604967.27 total 95.61 last 14.99) - open/close 1855 (time 1787604977.28 total 105.62 last 21.77) - open/close 2020 (time 1787604987.28 total 115.62 last 16.50) - open/close 2245 (time 1787604997.86 total 126.20 last 21.27) - open/close 2428 (time 1787605008.31 total 136.65 last 17.52) - open/close 2507 (time 1787605018.33 total 146.67 last 7.88) - open/close 2653 (time 1787605028.34 total 156.68 last 14.59) - open/close 2797 (time 1787605038.36 total 166.70 last 14.37) - open/close 2949 (time 1787605048.44 total 176.78 last 15.09) - open/close 3220 (time 1787605059.04 total 187.38 last 25.54) - open/close 3472 (time 1787605069.05 total 197.39 last 25.19) - open/close 3644 (time 1787605079.49 total 207.83 last 16.47) - open/close 3867 (time 1787605089.52 total 217.86 last 22.23) - open/close 3989 (time 1787605100.14 total 228.48 last 11.49) - open/close 4209 (time 1787605111.01 total 239.35 last 20.24) - open/close 4417 (time 1787605121.06 total 249.40 last 20.69) - open/close 4641 (time 1787605131.99 total 260.33 last 20.49) - open/close 4800 (time 1787605142.00 total 270.34 last 15.88) - open/close 4950 (time 1787605152.03 total 280.37 last 14.96) - open/close 5119 (time 1787605162.24 total 290.58 last 16.55) - open/close 5345 (time 1787605172.85 total 301.19 last 21.30) - open/close 5561 (time 1787605183.28 total 311.62 last 20.70) - open/close 5713 (time 1787605193.59 total 321.93 last 14.75) - open/close 5945 (time 1787605204.54 total 332.89 last 21.17) - open/close 6171 (time 1787605214.55 total 342.89 last 22.58) - open/close 6342 (time 1787605224.56 total 352.90 last 17.10) - open/close 6561 (time 1787605234.60 total 362.94 last 21.80) - open/close 6824 (time 1787605244.61 total 372.95 last 26.29) - open/close 6998 (time 1787605255.07 total 383.41 last 16.63) - open/close 7212 (time 1787605265.54 total 393.88 last 20.44) - open/close 7326 (time 1787605275.62 total 403.96 last 11.31) - open/close 7408 (time 1787605286.49 total 414.83 last 7.54) - open/close 7621 (time 1787605296.73 total 425.07 last 20.81) - open/close 7784 (time 1787605307.05 total 435.39 last 15.79) - open/close 7902 (time 1787605317.05 total 445.39 last 11.80) - open/close 8024 (time 1787605327.36 total 455.70 last 11.84) - open/close 8194 (time 1787605338.11 total 466.45 last 15.81) - open/close 8290 (time 1787605350.13 total 478.48 last 7.98) - open/close 8571 (time 1787605361.13 total 489.47 last 25.55) - open/close 8727 (time 1787605371.14 total 499.48 last 15.59) - open/close 8817 (time 1787605381.68 total 510.02 last 8.54) - open/close 8971 (time 1787605391.74 total 520.08 last 15.30) - open/close 9179 (time 1787605403.94 total 532.29 last 17.04) - open/close 9406 (time 1787605414.52 total 542.86 last 21.46) - open/close 9575 (time 1787605424.53 total 552.87 last 16.88) - open/close 9729 (time 1787605434.54 total 562.88 last 15.39) - open/close 9902 (time 1787605444.79 total 573.13 last 16.88) - open/close 10000 (time 1787605450.03 total 578.37 last 18.70) - open/close 10261 (time 1787605460.36 total 588.70 last 25.27) - open/close 10510 (time 1787605470.38 total 598.72 last 24.84) - open/close 10736 (time 1787605480.56 total 608.90 last 22.21) - open/close 10956 (time 1787605490.56 total 618.90 last 22.00) - open/close 11116 (time 1787605501.42 total 629.76 last 14.73) - open/close 11208 (time 1787605511.45 total 639.79 last 9.17) - open/close 11342 (time 1787605521.47 total 649.81 last 13.39) - open/close 11576 (time 1787605531.91 total 660.25 last 22.41) - open/close 11830 (time 1787605542.57 total 670.91 last 23.81) - open/close 12081 (time 1787605553.51 total 681.85 last 22.95) - open/close 12280 (time 1787605563.55 total 691.89 last 19.83) - open/close 12492 (time 1787605573.56 total 701.90 last 21.17) - open/close 12749 (time 1787605584.40 total 712.74 last 23.70) - open/close 12908 (time 1787605594.66 total 723.00 last 15.50) - open/close 13072 (time 1787605607.28 total 735.62 last 12.99) - open/close 13269 (time 1787605618.27 total 746.61 last 17.93) - open/close 13483 (time 1787605628.32 total 756.66 last 21.29) - open/close 13590 (time 1787605638.77 total 767.11 last 10.24) - open/close 13716 (time 1787605648.90 total 777.24 last 12.44) - open/close 13938 (time 1787605659.42 total 787.76 last 21.09) - open/close 14171 (time 1787605669.43 total 797.78 last 23.27) - open/close 14305 (time 1787605679.55 total 807.89 last 13.25) - open/close 14451 (time 1787605691.18 total 819.52 last 12.55) - open/close 14608 (time 1787605701.21 total 829.55 last 15.65) - open/close 14794 (time 1787605711.24 total 839.58 last 18.54) - open/close 14982 (time 1787605721.51 total 849.86 last 18.29) - open/close 15120 (time 1787605732.03 total 860.37 last 13.13) - open/close 15321 (time 1787605742.06 total 870.40 last 20.03) - open/close 15541 (time 1787605753.10 total 881.44 last 19.93) - open/close 15710 (time 1787605763.71 total 892.05 last 15.93) - open/close 15873 (time 1787605774.15 total 902.49 last 15.61) - open/close 16018 (time 1787605785.25 total 913.59 last 13.06) - open/close 16236 (time 1787605795.53 total 923.87 last 21.20) - open/close 16350 (time 1787605806.20 total 934.54 last 10.69) - open/close 16556 (time 1787605817.12 total 945.46 last 18.86) - open/close 16764 (time 1787605827.36 total 955.70 last 20.30) - open/close 16999 (time 1787605838.37 total 966.71 last 21.35) - open/close 17191 (time 1787605848.76 total 977.10 last 18.48) - open/close 17337 (time 1787605858.77 total 987.11 last 14.58) - open/close 17531 (time 1787605868.78 total 997.13 last 19.37) - open/close 17672 (time 1787605878.81 total 1007.15 last 14.07) - open/close 17888 (time 1787605888.99 total 1017.34 last 21.20) - open/close 18356 (time 1787605899.00 total 1027.34 last 46.77) - open/close 19150 (time 1787605909.01 total 1037.35 last 79.29) - open/close 19956 (time 1787605919.02 total 1047.36 last 80.54) - open/close 20701 (time 1787605929.06 total 1057.40 last 74.25) - open/close 21483 (time 1787605939.06 total 1067.40 last 78.17) - open/close 22315 (time 1787605949.06 total 1077.40 last 83.20) - open/close 23087 (time 1787605959.09 total 1087.43 last 77.00) - open/close 23865 (time 1787605969.09 total 1097.43 last 77.76) - open/close 24573 (time 1787605979.09 total 1107.43 last 70.79) - open/close 25207 (time 1787605989.10 total 1117.45 last 63.32) - open/close 25959 (time 1787605999.11 total 1127.45 last 75.16) - open/close 26780 (time 1787606009.11 total 1137.45 last 82.09) - open/close 27695 (time 1787606019.12 total 1147.46 last 91.46) - open/close 28530 (time 1787606029.14 total 1157.48 last 83.33) - open/close 29414 (time 1787606039.15 total 1167.49 last 88.31) - open/close 30000 (time 1787606044.71 total 1173.05 last 105.27) - open/close 30864 (time 1787606054.71 total 1183.06 last 86.39) - open/close 31724 (time 1787606064.72 total 1193.06 last 85.96) - open/close 32511 (time 1787606074.72 total 1203.06 last 78.70) - open/close 33383 (time 1787606084.74 total 1213.08 last 87.02) - open/close 34277 (time 1787606094.74 total 1223.08 last 89.39) - open/close 35154 (time 1787606104.75 total 1233.09 last 87.61) - open/close 35956 (time 1787606114.76 total 1243.10 last 80.13) - open/close 36657 (time 1787606124.78 total 1253.12 last 69.99) - open/close 37130 (time 1787606134.78 total 1263.12 last 47.27) - open/close 37694 (time 1787606144.80 total 1273.14 last 56.33) - open/close 38436 (time 1787606154.80 total 1283.14 last 74.17) - open/close 39216 (time 1787606164.82 total 1293.16 last 77.87) - open/close 39979 (time 1787606174.82 total 1303.16 last 76.27) - open/close 40687 (time 1787606184.82 total 1313.16 last 70.79) - open/close 41459 (time 1787606194.84 total 1323.18 last 77.08) - open/close 42146 (time 1787606204.86 total 1333.20 last 68.57) - open/close 42899 (time 1787606214.86 total 1343.20 last 75.28) - open/close 43659 (time 1787606224.86 total 1353.20 last 75.96) - open/close 44480 (time 1787606234.87 total 1363.21 last 82.02) - open/close 45305 (time 1787606244.88 total 1373.22 last 82.48) - open/close 46061 (time 1787606254.89 total 1383.23 last 75.51) - open/close 46892 (time 1787606264.90 total 1393.24 last 82.97) - open/close 47814 (time 1787606274.91 total 1403.25 last 92.15) - open/close 48687 (time 1787606284.92 total 1413.26 last 87.21) - open/close 49395 (time 1787606294.92 total 1423.26 last 70.77) total: 50000 open/close in 1432.57 seconds: 34.90 ops/second - unlinked 0 (time 1787606309 ; total 0 ; last 0) - unlinked 10000 (time 1787606426 ; total 117 ; last 117) - unlinked 20000 (time 1787606540 ; total 231 ; last 114) - unlinked 30000 (time 1787606669 ; total 360 ; last 129) - unlinked 40000 (time 1787606792 ; total 483 ; last 123) total: 50000 unlinks in 613 seconds: 81.566071 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server unloading modules via unload_modules_local on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local oleg233-server: modules unloaded. pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2