== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:54:22 (1787871262) start mds service on oleg630-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg630-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg630-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1787871346.13 total 12.07 last 18.89) - open/close 473 (time 1787871356.49 total 22.43 last 23.65) - open/close 723 (time 1787871366.50 total 32.44 last 24.97) - open/close 823 (time 1787871377.04 total 42.97 last 9.49) - open/close 1025 (time 1787871387.55 total 53.49 last 19.21) - open/close 1267 (time 1787871397.72 total 63.66 last 23.80) - open/close 1452 (time 1787871408.04 total 73.98 last 17.92) - open/close 1679 (time 1787871418.39 total 84.33 last 21.94) - open/close 1901 (time 1787871428.87 total 94.81 last 21.18) - open/close 2108 (time 1787871439.13 total 105.06 last 20.19) - open/close 2348 (time 1787871449.13 total 115.07 last 23.99) - open/close 2557 (time 1787871459.13 total 125.07 last 20.90) - open/close 2769 (time 1787871469.16 total 135.10 last 21.14) - open/close 3042 (time 1787871480.08 total 146.02 last 24.99) - open/close 3265 (time 1787871490.14 total 156.07 last 22.18) - open/close 3548 (time 1787871500.76 total 166.70 last 26.63) - open/close 3722 (time 1787871510.78 total 176.71 last 17.38) - open/close 3894 (time 1787871520.78 total 186.72 last 17.18) - open/close 4075 (time 1787871531.62 total 197.55 last 16.71) - open/close 4319 (time 1787871542.29 total 208.23 last 22.85) - open/close 4536 (time 1787871552.73 total 218.67 last 20.79) - open/close 4765 (time 1787871562.74 total 228.68 last 22.89) - open/close 4995 (time 1787871572.75 total 238.68 last 22.99) - open/close 5264 (time 1787871583.41 total 249.34 last 25.23) - open/close 5445 (time 1787871593.57 total 259.50 last 17.82) - open/close 5630 (time 1787871603.93 total 269.87 last 17.85) - open/close 5744 (time 1787871613.94 total 279.87 last 11.39) - open/close 5917 (time 1787871624.09 total 290.02 last 17.04) - open/close 6084 (time 1787871634.10 total 300.04 last 16.67) - open/close 6255 (time 1787871644.10 total 310.04 last 17.10) - open/close 6372 (time 1787871655.12 total 321.06 last 10.62) - open/close 6600 (time 1787871665.30 total 331.23 last 22.40) - open/close 6815 (time 1787871675.30 total 341.24 last 21.49) - open/close 7053 (time 1787871685.97 total 351.91 last 22.31) - open/close 7290 (time 1787871696.09 total 362.02 last 23.43) - open/close 7522 (time 1787871706.15 total 372.08 last 23.06) - open/close 7737 (time 1787871716.37 total 382.30 last 21.03) - open/close 7974 (time 1787871726.49 total 392.42 last 23.42) - open/close 8192 (time 1787871736.50 total 402.44 last 21.77) - open/close 8418 (time 1787871746.52 total 412.46 last 22.55) - open/close 8624 (time 1787871757.33 total 423.27 last 19.05) - open/close 8728 (time 1787871770.08 total 436.01 last 8.16) - open/close 8891 (time 1787871780.31 total 446.25 last 15.92) - open/close 9117 (time 1787871790.55 total 456.49 last 22.08) - open/close 9335 (time 1787871800.56 total 466.49 last 21.79) - open/close 9578 (time 1787871810.96 total 476.90 last 23.35) - open/close 9808 (time 1787871820.97 total 486.90 last 22.99) - open/close 10000 (time 1787871829.69 total 495.62 last 22.02) - open/close 10225 (time 1787871840.62 total 506.56 last 20.57) - open/close 10402 (time 1787871850.81 total 516.75 last 17.37) - open/close 10532 (time 1787871861.88 total 527.81 last 11.75) - open/close 10732 (time 1787871876.96 total 542.89 last 13.26) - open/close 10913 (time 1787871886.98 total 552.91 last 18.07) - open/close 11058 (time 1787871898.15 total 564.08 last 12.98) - open/close 11328 (time 1787871908.37 total 574.31 last 26.40) - open/close 11485 (time 1787871921.00 total 586.93 last 12.44) - open/close 11730 (time 1787871931.00 total 596.94 last 24.49) - open/close 11978 (time 1787871941.21 total 607.14 last 24.30) - open/close 12254 (time 1787871951.21 total 617.15 last 27.58) - open/close 12470 (time 1787871962.14 total 628.07 last 19.77) - open/close 12676 (time 1787871972.14 total 638.07 last 20.60) - open/close 12779 (time 1787871982.76 total 648.69 last 9.70) - open/close 12980 (time 1787871993.50 total 659.44 last 18.71) - open/close 13219 (time 1787872003.50 total 669.44 last 23.90) - open/close 13364 (time 1787872013.52 total 679.45 last 14.48) - open/close 13515 (time 1787872023.63 total 689.57 last 14.93) - open/close 13736 (time 1787872033.85 total 699.78 last 21.63) - open/close 13862 (time 1787872044.00 total 709.93 last 12.42) - open/close 14115 (time 1787872054.90 total 720.83 last 23.21) - open/close 14328 (time 1787872065.56 total 731.50 last 19.97) - open/close 14574 (time 1787872076.51 total 742.44 last 22.47) - open/close 14768 (time 1787872086.51 total 752.45 last 19.39) - open/close 15018 (time 1787872096.98 total 762.91 last 23.89) - open/close 15246 (time 1787872107.29 total 773.22 last 22.12) - open/close 15456 (time 1787872117.36 total 783.29 last 20.85) - open/close 15701 (time 1787872127.83 total 793.76 last 23.40) - open/close 15935 (time 1787872138.41 total 804.34 last 22.12) - open/close 16207 (time 1787872148.42 total 814.35 last 27.17) - open/close 16461 (time 1787872158.43 total 824.36 last 25.37) - open/close 16735 (time 1787872168.87 total 834.80 last 26.25) - open/close 16926 (time 1787872178.96 total 844.89 last 18.92) - open/close 17180 (time 1787872188.96 total 854.90 last 25.40) - open/close 17354 (time 1787872199.71 total 865.65 last 16.19) - open/close 17524 (time 1787872209.73 total 875.67 last 16.96) - open/close 17652 (time 1787872220.17 total 886.11 last 12.26) - open/close 17753 (time 1787872233.12 total 899.06 last 7.80) - open/close 17905 (time 1787872244.66 total 910.59 last 13.18) - open/close 18068 (time 1787872254.66 total 920.59 last 16.30) - open/close 18293 (time 1787872264.67 total 930.60 last 22.48) - open/close 18925 (time 1787872274.67 total 940.61 last 63.16) - open/close 19551 (time 1787872284.68 total 950.61 last 62.56) - open/close 20000 (time 1787872291.33 total 957.26 last 67.51) - open/close 20637 (time 1787872301.33 total 967.27 last 63.69) - open/close 21286 (time 1787872311.34 total 977.27 last 64.86) - open/close 21917 (time 1787872321.35 total 987.29 last 62.99) - open/close 22616 (time 1787872331.37 total 997.30 last 69.80) - open/close 23361 (time 1787872341.37 total 1007.30 last 74.50) - open/close 24066 (time 1787872351.39 total 1017.32 last 70.37) - open/close 24765 (time 1787872361.39 total 1027.32 last 69.89) - open/close 25415 (time 1787872371.40 total 1037.34 last 64.89) - open/close 26046 (time 1787872381.42 total 1047.36 last 62.98) - open/close 26733 (time 1787872391.43 total 1057.36 last 68.67) - open/close 27450 (time 1787872401.43 total 1067.37 last 71.67) - open/close 28010 (time 1787872411.47 total 1077.41 last 55.78) - open/close 28663 (time 1787872421.50 total 1087.44 last 65.11) - open/close 29324 (time 1787872431.51 total 1097.44 last 66.07) - open/close 30000 (time 1787872441.38 total 1107.32 last 68.43) - open/close 30748 (time 1787872451.39 total 1117.33 last 74.76) - open/close 31489 (time 1787872461.40 total 1127.33 last 74.05) - open/close 32170 (time 1787872471.42 total 1137.35 last 67.96) - open/close 32896 (time 1787872481.42 total 1147.35 last 72.58) - open/close 33555 (time 1787872491.42 total 1157.36 last 65.89) - open/close 34228 (time 1787872501.44 total 1167.37 last 67.19) - open/close 34892 (time 1787872511.44 total 1177.37 last 66.39) - open/close 35489 (time 1787872521.44 total 1187.38 last 59.69) - open/close 36173 (time 1787872531.48 total 1197.41 last 68.15) - open/close 36786 (time 1787872541.48 total 1207.41 last 61.29) - open/close 37405 (time 1787872551.50 total 1217.44 last 61.77) - open/close 38110 (time 1787872561.50 total 1227.44 last 70.48) - open/close 38735 (time 1787872571.52 total 1237.45 last 62.41) - open/close 39344 (time 1787872581.52 total 1247.46 last 60.88) - open/close 39897 (time 1787872591.53 total 1257.46 last 55.27) - open/close 40545 (time 1787872601.53 total 1267.47 last 64.76) - open/close 41172 (time 1787872611.54 total 1277.47 last 62.67) - open/close 41828 (time 1787872621.54 total 1287.47 last 65.58) - open/close 42554 (time 1787872631.55 total 1297.49 last 72.49) - open/close 43224 (time 1787872641.55 total 1307.49 last 66.99) - open/close 43776 (time 1787872651.56 total 1317.49 last 55.18) - open/close 44493 (time 1787872661.57 total 1327.50 last 71.64) - open/close 45186 (time 1787872671.57 total 1337.50 last 69.29) - open/close 46007 (time 1787872681.57 total 1347.51 last 82.05) - open/close 46719 (time 1787872691.58 total 1357.52 last 71.16) - open/close 47385 (time 1787872701.58 total 1367.52 last 66.58) - open/close 48041 (time 1787872711.59 total 1377.53 last 65.54) - open/close 48722 (time 1787872721.59 total 1387.53 last 68.09) - open/close 49428 (time 1787872731.62 total 1397.55 last 70.41) total: 50000 open/close in 1406.77 seconds: 35.54 ops/second - unlinked 0 (time 1787872745 ; total 0 ; last 0) - unlinked 10000 (time 1787872905 ; total 160 ; last 160) - unlinked 20000 (time 1787873067 ; total 322 ; last 162) - unlinked 30000 (time 1787873214 ; total 469 ; last 147) - unlinked 40000 (time 1787873363 ; total 618 ; last 149) total: 50000 unlinks in 769 seconds: 65.019508 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: modules unloaded. pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2