== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:20:23 (1775766023) start mds service on oleg656-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing load_modules_local oleg656-server: Loading modules from /home/green/git/lustre-release/lustre oleg656-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg656-server: detected 4 online CPUs by sysfs oleg656-server: MODOPTS_LIBCFS= oleg656-server: Force libcfs to create 2 CPU partitions oleg656-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg656-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg656-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg656-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1775766105.03 total 10.11 last 19.39) - open/close 470 (time 1775766115.04 total 20.12 last 27.37) - open/close 665 (time 1775766125.04 total 30.12 last 19.49) - open/close 703 (time 1775766135.41 total 40.49 last 3.67) - open/close 932 (time 1775766145.83 total 50.90 last 21.98) - open/close 1079 (time 1775766155.94 total 61.02 last 14.53) - open/close 1242 (time 1775766166.43 total 71.51 last 15.54) - open/close 1404 (time 1775766177.19 total 82.27 last 15.06) - open/close 1650 (time 1775766188.15 total 93.23 last 22.44) - open/close 1904 (time 1775766198.15 total 103.23 last 25.39) - open/close 1988 (time 1775766208.16 total 113.23 last 8.40) - open/close 2069 (time 1775766218.16 total 123.23 last 8.10) - open/close 2277 (time 1775766229.73 total 134.80 last 17.98) - open/close 2519 (time 1775766240.16 total 145.24 last 23.19) - open/close 2689 (time 1775766250.17 total 155.25 last 16.98) - open/close 2843 (time 1775766262.04 total 167.12 last 12.98) - open/close 3012 (time 1775766272.09 total 177.17 last 16.81) - open/close 3207 (time 1775766282.28 total 187.36 last 19.14) - open/close 3405 (time 1775766292.31 total 197.39 last 19.75) - open/close 3549 (time 1775766304.57 total 209.65 last 11.75) - open/close 3732 (time 1775766314.59 total 219.67 last 18.27) - open/close 3879 (time 1775766327.37 total 232.45 last 11.50) - open/close 3988 (time 1775766338.03 total 243.10 last 10.23) - open/close 4129 (time 1775766348.06 total 253.14 last 14.05) - open/close 4297 (time 1775766358.08 total 263.15 last 16.77) - open/close 4396 (time 1775766368.63 total 273.71 last 9.38) - open/close 4469 (time 1775766380.80 total 285.87 last 6.00) - open/close 4584 (time 1775766390.80 total 295.88 last 11.49) - open/close 4679 (time 1775766401.26 total 306.34 last 9.09) - open/close 4815 (time 1775766411.27 total 316.34 last 13.59) - open/close 4923 (time 1775766422.25 total 327.32 last 9.84) - open/close 4985 (time 1775766432.25 total 337.33 last 6.20) - open/close 5155 (time 1775766442.27 total 347.35 last 16.97) - open/close 5397 (time 1775766452.29 total 357.36 last 24.17) - open/close 5545 (time 1775766462.92 total 368.00 last 13.92) - open/close 5752 (time 1775766473.42 total 378.50 last 19.71) - open/close 5864 (time 1775766484.46 total 389.54 last 10.14) - open/close 5930 (time 1775766495.34 total 400.42 last 6.06) - open/close 6106 (time 1775766505.35 total 410.42 last 17.59) - open/close 6339 (time 1775766515.43 total 420.51 last 23.11) - open/close 6481 (time 1775766526.51 total 431.59 last 12.81) - open/close 6659 (time 1775766539.04 total 444.12 last 14.20) - open/close 6837 (time 1775766549.08 total 454.15 last 17.74) - open/close 7106 (time 1775766559.10 total 464.18 last 26.83) - open/close 7381 (time 1775766569.51 total 474.59 last 26.41) - open/close 7616 (time 1775766580.08 total 485.16 last 22.24) - open/close 7803 (time 1775766590.10 total 495.18 last 18.67) - open/close 8003 (time 1775766600.82 total 505.89 last 18.66) - open/close 8111 (time 1775766611.50 total 516.58 last 10.11) - open/close 8206 (time 1775766623.03 total 528.10 last 8.24) - open/close 8455 (time 1775766633.93 total 539.01 last 22.83) - open/close 8685 (time 1775766643.95 total 549.03 last 22.96) - open/close 8877 (time 1775766654.63 total 559.71 last 17.98) - open/close 9096 (time 1775766664.74 total 569.81 last 21.67) - open/close 9347 (time 1775766675.58 total 580.65 last 23.16) - open/close 9525 (time 1775766685.60 total 590.68 last 17.75) - open/close 9738 (time 1775766696.17 total 601.25 last 20.16) - open/close 10000 (time 1775766704.89 total 609.97 last 30.02) - open/close 10137 (time 1775766717.90 total 622.98 last 10.53) - open/close 10358 (time 1775766730.09 total 635.16 last 18.14) - open/close 10559 (time 1775766740.40 total 645.48 last 19.49) - open/close 10674 (time 1775766750.44 total 655.51 last 11.46) - open/close 10792 (time 1775766761.31 total 666.39 last 10.85) - open/close 10949 (time 1775766771.37 total 676.44 last 15.61) - open/close 11115 (time 1775766781.38 total 686.46 last 16.58) - open/close 11356 (time 1775766792.30 total 697.37 last 22.07) - open/close 11584 (time 1775766802.46 total 707.54 last 22.43) - open/close 11736 (time 1775766812.90 total 717.98 last 14.56) - open/close 11930 (time 1775766823.69 total 728.76 last 17.99) - open/close 12133 (time 1775766833.96 total 739.03 last 19.76) - open/close 12360 (time 1775766844.89 total 749.97 last 20.76) - open/close 12583 (time 1775766855.39 total 760.46 last 21.25) - open/close 12814 (time 1775766865.39 total 770.47 last 23.08) - open/close 12940 (time 1775766875.43 total 780.51 last 12.55) - open/close 13238 (time 1775766886.31 total 791.38 last 27.41) - open/close 13449 (time 1775766896.95 total 802.03 last 19.82) - open/close 13711 (time 1775766906.97 total 812.05 last 26.16) - open/close 13915 (time 1775766917.50 total 822.58 last 19.36) - open/close 14149 (time 1775766929.15 total 834.23 last 20.09) - open/close 14344 (time 1775766939.68 total 844.76 last 18.52) - open/close 14542 (time 1775766949.98 total 855.06 last 19.22) - open/close 14742 (time 1775766960.00 total 865.08 last 19.97) - open/close 14986 (time 1775766970.82 total 875.90 last 22.54) - open/close 15124 (time 1775766980.85 total 885.92 last 13.77) - open/close 15260 (time 1775766991.35 total 896.43 last 12.94) - open/close 15440 (time 1775767001.39 total 906.46 last 17.94) - open/close 15647 (time 1775767011.96 total 917.04 last 19.57) - open/close 15748 (time 1775767022.43 total 927.51 last 9.64) - open/close 15933 (time 1775767032.44 total 937.52 last 18.49) - open/close 16129 (time 1775767043.81 total 948.89 last 17.24) - open/close 16401 (time 1775767054.82 total 959.90 last 24.69) - open/close 16654 (time 1775767065.78 total 970.86 last 23.10) - open/close 16873 (time 1775767075.79 total 980.87 last 21.87) - open/close 17046 (time 1775767085.80 total 990.88 last 17.28) - open/close 17268 (time 1775767096.04 total 1001.12 last 21.68) - open/close 17476 (time 1775767106.04 total 1011.12 last 20.80) - open/close 18173 (time 1775767116.05 total 1021.12 last 69.68) - open/close 18900 (time 1775767126.06 total 1031.14 last 72.59) - open/close 19567 (time 1775767136.08 total 1041.16 last 66.58) - open/close 20000 (time 1775767142.12 total 1047.20 last 71.69) - open/close 20684 (time 1775767152.12 total 1057.20 last 68.39) - open/close 21380 (time 1775767162.13 total 1067.20 last 69.56) - open/close 22045 (time 1775767172.13 total 1077.21 last 66.47) - open/close 22709 (time 1775767182.13 total 1087.21 last 66.39) - open/close 23317 (time 1775767192.14 total 1097.22 last 60.75) - open/close 24000 (time 1775767202.16 total 1107.24 last 68.16) - open/close 24705 (time 1775767212.17 total 1117.24 last 70.46) - open/close 25296 (time 1775767222.18 total 1127.26 last 59.00) - open/close 25985 (time 1775767232.19 total 1137.26 last 68.87) - open/close 26709 (time 1775767242.19 total 1147.27 last 72.38) - open/close 27392 (time 1775767252.21 total 1157.29 last 68.15) - open/close 28081 (time 1775767262.22 total 1167.30 last 68.84) - open/close 28753 (time 1775767272.23 total 1177.31 last 67.14) - open/close 29447 (time 1775767282.24 total 1187.31 last 69.35) - open/close 30000 (time 1775767289.44 total 1194.51 last 76.81) - open/close 30682 (time 1775767299.44 total 1204.51 last 68.19) - open/close 31401 (time 1775767309.44 total 1214.52 last 71.90) - open/close 32064 (time 1775767319.44 total 1224.52 last 66.27) - open/close 32761 (time 1775767329.45 total 1234.52 last 69.67) - open/close 33457 (time 1775767339.45 total 1244.53 last 69.57) - open/close 34146 (time 1775767349.45 total 1254.53 last 68.87) - open/close 34896 (time 1775767359.47 total 1264.54 last 74.91) - open/close 35628 (time 1775767369.47 total 1274.55 last 73.17) - open/close 36328 (time 1775767379.48 total 1284.56 last 69.93) - open/close 37022 (time 1775767389.48 total 1294.56 last 69.38) - open/close 37729 (time 1775767399.50 total 1304.58 last 70.59) - open/close 38379 (time 1775767409.51 total 1314.59 last 64.93) - open/close 39052 (time 1775767419.52 total 1324.60 last 67.24) - open/close 39724 (time 1775767429.52 total 1334.60 last 67.19) - open/close 40000 (time 1775767433.49 total 1338.57 last 69.57) - open/close 40673 (time 1775767443.49 total 1348.57 last 67.27) - open/close 41398 (time 1775767453.49 total 1358.57 last 72.50) - open/close 42077 (time 1775767463.52 total 1368.60 last 67.71) - open/close 42639 (time 1775767473.54 total 1378.62 last 56.11) - open/close 43290 (time 1775767483.56 total 1388.63 last 64.98) - open/close 43956 (time 1775767493.56 total 1398.63 last 66.60) - open/close 44623 (time 1775767503.56 total 1408.63 last 66.70) - open/close 45289 (time 1775767513.56 total 1418.63 last 66.60) - open/close 45976 (time 1775767523.56 total 1428.64 last 68.66) - open/close 46669 (time 1775767533.57 total 1438.65 last 69.24) - open/close 47277 (time 1775767543.58 total 1448.66 last 60.74) - open/close 47843 (time 1775767553.58 total 1458.66 last 56.59) - open/close 48441 (time 1775767563.59 total 1468.67 last 59.75) - open/close 49105 (time 1775767573.61 total 1478.69 last 66.25) - open/close 49787 (time 1775767583.61 total 1488.69 last 68.20) total: 50000 open/close in 1491.55 seconds: 33.52 ops/second - unlinked 0 (time 1775767591 ; total 0 ; last 0) - unlinked 10000 (time 1775767753 ; total 162 ; last 162) - unlinked 20000 (time 1775767915 ; total 324 ; last 162) - unlinked 30000 (time 1775768072 ; total 481 ; last 157) - unlinked 40000 (time 1775768229 ; total 638 ; last 157) total: 50000 unlinks in 789 seconds: 63.371357 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg656-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg656-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg656-server unloading modules via unload_modules_local on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing unload_modules_local oleg656-server: modules unloaded. pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2