== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:41:36 (1787740896) start mds service on oleg627-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: detected 4 online CPUs by sysfs oleg627-server: MODOPTS_LIBCFS= oleg627-server: Force libcfs to create 2 CPU partitions oleg627-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg627-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg627-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 167 (time 1787740977.05 total 12.77 last 13.08) - open/close 437 (time 1787740987.54 total 23.26 last 25.72) - open/close 637 (time 1787740998.13 total 33.85 last 18.90) - open/close 960 (time 1787741008.99 total 44.71 last 29.73) - open/close 1212 (time 1787741019.00 total 54.72 last 25.19) - open/close 1432 (time 1787741029.56 total 65.28 last 20.84) - open/close 1675 (time 1787741041.64 total 77.36 last 20.11) - open/close 1793 (time 1787741051.67 total 87.39 last 11.76) - open/close 1948 (time 1787741061.70 total 97.42 last 15.45) - open/close 2241 (time 1787741071.71 total 107.43 last 29.28) - open/close 2521 (time 1787741082.67 total 118.39 last 25.55) - open/close 2715 (time 1787741092.69 total 128.41 last 19.36) - open/close 2921 (time 1787741102.80 total 138.52 last 20.38) - open/close 3145 (time 1787741112.84 total 148.56 last 22.31) - open/close 3381 (time 1787741122.85 total 158.57 last 23.57) - open/close 3686 (time 1787741133.65 total 169.37 last 28.25) - open/close 3890 (time 1787741143.69 total 179.41 last 20.33) - open/close 4077 (time 1787741153.71 total 189.43 last 18.66) - open/close 4181 (time 1787741164.06 total 199.78 last 10.05) - open/close 4283 (time 1787741174.81 total 210.53 last 9.49) - open/close 4470 (time 1787741184.84 total 220.56 last 18.64) - open/close 4742 (time 1787741195.41 total 231.13 last 25.74) - open/close 5080 (time 1787741206.36 total 242.08 last 30.86) - open/close 5343 (time 1787741216.80 total 252.52 last 25.21) - open/close 5582 (time 1787741226.80 total 262.52 last 23.90) - open/close 5790 (time 1787741237.03 total 272.75 last 20.33) - open/close 6079 (time 1787741247.04 total 282.76 last 28.88) - open/close 6316 (time 1787741257.04 total 292.76 last 23.68) - open/close 6551 (time 1787741267.07 total 302.79 last 23.44) - open/close 6707 (time 1787741278.00 total 313.72 last 14.28) - open/close 6943 (time 1787741288.02 total 323.74 last 23.56) - open/close 7133 (time 1787741298.27 total 333.98 last 18.54) - open/close 7398 (time 1787741308.28 total 343.99 last 26.47) - open/close 7636 (time 1787741318.29 total 354.01 last 23.76) - open/close 7882 (time 1787741329.04 total 364.76 last 22.89) - open/close 8025 (time 1787741339.35 total 375.07 last 13.87) - open/close 8154 (time 1787741349.59 total 385.31 last 12.60) - open/close 8391 (time 1787741359.71 total 395.43 last 23.43) - open/close 8600 (time 1787741370.65 total 406.37 last 19.10) - open/close 8814 (time 1787741380.96 total 416.68 last 20.74) - open/close 9006 (time 1787741391.58 total 427.30 last 18.08) - open/close 9221 (time 1787741401.60 total 437.31 last 21.47) - open/close 9463 (time 1787741411.60 total 447.32 last 24.18) - open/close 9612 (time 1787741422.36 total 458.07 last 13.86) - open/close 9850 (time 1787741432.38 total 468.10 last 23.73) - open/close 10000 (time 1787741440.89 total 476.61 last 17.64) - open/close 10120 (time 1787741451.88 total 487.60 last 10.92) - open/close 10224 (time 1787741462.18 total 497.89 last 10.10) - open/close 10402 (time 1787741472.18 total 507.90 last 17.79) - open/close 10552 (time 1787741482.66 total 518.38 last 14.32) - open/close 10783 (time 1787741493.69 total 529.40 last 20.95) - open/close 10911 (time 1787741503.73 total 539.45 last 12.74) - open/close 11039 (time 1787741514.79 total 550.51 last 11.57) - open/close 11318 (time 1787741524.79 total 560.51 last 27.90) - open/close 11389 (time 1787741535.86 total 571.58 last 6.41) - open/close 11550 (time 1787741545.87 total 581.59 last 16.09) - open/close 11774 (time 1787741555.88 total 591.60 last 22.38) - open/close 11970 (time 1787741566.26 total 601.98 last 18.88) - open/close 12125 (time 1787741577.27 total 612.99 last 14.08) - open/close 12362 (time 1787741587.27 total 622.99 last 23.69) - open/close 12569 (time 1787741598.18 total 633.90 last 18.98) - open/close 12780 (time 1787741608.81 total 644.53 last 19.84) - open/close 13039 (time 1787741618.82 total 654.54 last 25.89) - open/close 13293 (time 1787741628.82 total 664.54 last 25.39) - open/close 13534 (time 1787741638.82 total 674.54 last 24.10) - open/close 13788 (time 1787741648.83 total 684.55 last 25.38) - open/close 13927 (time 1787741659.71 total 695.43 last 12.77) - open/close 14136 (time 1787741669.73 total 705.45 last 20.87) - open/close 14432 (time 1787741679.74 total 715.46 last 29.55) - open/close 14699 (time 1787741690.33 total 726.05 last 25.23) - open/close 14922 (time 1787741700.35 total 736.07 last 22.24) - open/close 15207 (time 1787741711.22 total 746.94 last 26.23) - open/close 15415 (time 1787741721.63 total 757.35 last 19.98) - open/close 15616 (time 1787741734.59 total 770.31 last 15.51) - open/close 15758 (time 1787741744.61 total 780.33 last 14.17) - open/close 15952 (time 1787741755.42 total 791.14 last 17.94) - open/close 16060 (time 1787741766.08 total 801.80 last 10.14) - open/close 16238 (time 1787741776.10 total 811.82 last 17.76) - open/close 16390 (time 1787741786.48 total 822.20 last 14.65) - open/close 16572 (time 1787741797.54 total 833.26 last 16.45) - open/close 16804 (time 1787741808.09 total 843.81 last 21.99) - open/close 16904 (time 1787741818.72 total 854.44 last 9.40) - open/close 17020 (time 1787741828.73 total 864.45 last 11.59) - open/close 17133 (time 1787741839.41 total 875.13 last 10.58) - open/close 17260 (time 1787741849.45 total 885.17 last 12.66) - open/close 17467 (time 1787741859.46 total 895.18 last 20.68) - open/close 17692 (time 1787741869.48 total 905.20 last 22.46) - open/close 17854 (time 1787741882.47 total 918.19 last 12.47) - open/close 17952 (time 1787741893.93 total 929.65 last 8.55) - open/close 18182 (time 1787741904.11 total 939.83 last 22.60) - open/close 18406 (time 1787741914.16 total 949.88 last 22.28) - open/close 18921 (time 1787741924.16 total 959.88 last 51.48) - open/close 19324 (time 1787741934.18 total 969.90 last 40.24) - open/close 19819 (time 1787741944.19 total 979.91 last 49.46) - open/close 20000 (time 1787741947.77 total 983.49 last 50.47) - open/close 20690 (time 1787741957.79 total 993.51 last 68.90) - open/close 21403 (time 1787741967.79 total 1003.51 last 71.26) - open/close 22249 (time 1787741977.80 total 1013.52 last 84.57) - open/close 23047 (time 1787741987.80 total 1023.52 last 79.76) - open/close 23919 (time 1787741997.83 total 1033.55 last 86.97) - open/close 24709 (time 1787742007.83 total 1043.55 last 78.98) - open/close 25549 (time 1787742017.84 total 1053.56 last 83.96) - open/close 26406 (time 1787742027.84 total 1063.56 last 85.64) - open/close 27315 (time 1787742037.84 total 1073.56 last 90.90) - open/close 28131 (time 1787742047.85 total 1083.57 last 81.56) - open/close 28742 (time 1787742057.86 total 1093.58 last 61.04) - open/close 29531 (time 1787742067.88 total 1103.60 last 78.75) - open/close 30000 (time 1787742073.70 total 1109.42 last 80.51) - open/close 30758 (time 1787742083.71 total 1119.43 last 75.77) - open/close 31571 (time 1787742093.71 total 1129.43 last 81.26) - open/close 32366 (time 1787742103.72 total 1139.44 last 79.46) - open/close 33142 (time 1787742113.72 total 1149.44 last 77.56) - open/close 33965 (time 1787742123.72 total 1159.44 last 82.29) - open/close 34823 (time 1787742133.73 total 1169.45 last 85.73) - open/close 35620 (time 1787742143.75 total 1179.47 last 79.56) - open/close 36451 (time 1787742153.75 total 1189.47 last 83.09) - open/close 37311 (time 1787742163.76 total 1199.48 last 85.92) - open/close 38141 (time 1787742173.76 total 1209.48 last 82.95) - open/close 38892 (time 1787742183.79 total 1219.51 last 74.90) - open/close 39705 (time 1787742193.80 total 1229.52 last 81.26) - open/close 40000 (time 1787742198.00 total 1233.72 last 70.20) - open/close 40646 (time 1787742208.01 total 1243.73 last 64.51) - open/close 41369 (time 1787742218.05 total 1253.77 last 72.06) - open/close 42135 (time 1787742228.05 total 1263.77 last 76.58) - open/close 42791 (time 1787742238.10 total 1273.82 last 65.26) - open/close 43478 (time 1787742248.10 total 1283.82 last 68.68) - open/close 44103 (time 1787742258.11 total 1293.83 last 62.48) - open/close 44857 (time 1787742268.11 total 1303.83 last 75.38) - open/close 45634 (time 1787742278.12 total 1313.84 last 77.64) - open/close 46324 (time 1787742288.12 total 1323.84 last 69.00) - open/close 46992 (time 1787742298.14 total 1333.86 last 66.66) - open/close 47609 (time 1787742308.15 total 1343.87 last 61.63) - open/close 48231 (time 1787742318.16 total 1353.88 last 62.13) - open/close 48799 (time 1787742328.17 total 1363.88 last 56.76) - open/close 49609 (time 1787742338.17 total 1373.89 last 80.96) total: 50000 open/close in 1379.21 seconds: 36.25 ops/second - unlinked 0 (time 1787742347 ; total 0 ; last 0) - unlinked 10000 (time 1787742488 ; total 141 ; last 141) - unlinked 20000 (time 1787742624 ; total 277 ; last 136) - unlinked 30000 (time 1787742774 ; total 427 ; last 150) - unlinked 40000 (time 1787742919 ; total 572 ; last 145) total: 50000 unlinks in 719 seconds: 69.541031 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server unloading modules via unload_modules_local on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing unload_modules_local oleg627-server: modules unloaded. pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2