== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:44:53 (1786308293) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 223 (time 1786308387.87 total 10.39 last 21.45) - open/close 486 (time 1786308398.87 total 21.40 last 23.91) - open/close 734 (time 1786308409.45 total 31.98 last 23.44) - open/close 928 (time 1786308419.47 total 42.00 last 19.36) - open/close 1193 (time 1786308429.48 total 52.01 last 26.48) - open/close 1374 (time 1786308439.49 total 62.02 last 18.08) - open/close 1631 (time 1786308450.21 total 72.74 last 23.97) - open/close 1887 (time 1786308460.23 total 82.76 last 25.55) - open/close 2082 (time 1786308471.25 total 93.78 last 17.70) - open/close 2293 (time 1786308481.55 total 104.08 last 20.48) - open/close 2507 (time 1786308492.52 total 115.04 last 19.51) - open/close 2638 (time 1786308502.61 total 125.13 last 12.98) - open/close 2878 (time 1786308512.62 total 135.14 last 23.98) - open/close 3114 (time 1786308522.62 total 145.14 last 23.60) - open/close 3362 (time 1786308532.62 total 155.15 last 24.79) - open/close 3488 (time 1786308543.33 total 165.86 last 11.77) - open/close 3732 (time 1786308553.35 total 175.88 last 24.35) - open/close 3935 (time 1786308565.23 total 187.76 last 17.09) - open/close 4107 (time 1786308575.24 total 197.77 last 17.19) - open/close 4296 (time 1786308585.46 total 207.99 last 18.49) - open/close 4473 (time 1786308595.93 total 218.46 last 16.90) - open/close 4621 (time 1786308605.96 total 228.49 last 14.76) - open/close 4700 (time 1786308615.97 total 238.50 last 7.89) - open/close 4839 (time 1786308625.98 total 248.50 last 13.89) - open/close 5064 (time 1786308636.28 total 258.81 last 21.83) - open/close 5303 (time 1786308647.20 total 269.72 last 21.90) - open/close 5417 (time 1786308657.87 total 280.40 last 10.68) - open/close 5570 (time 1786308667.88 total 290.41 last 15.29) - open/close 5787 (time 1786308677.92 total 300.45 last 21.61) - open/close 6023 (time 1786308687.93 total 310.46 last 23.58) - open/close 6245 (time 1786308697.93 total 320.46 last 22.19) - open/close 6490 (time 1786308708.11 total 330.64 last 24.08) - open/close 6658 (time 1786308720.53 total 343.06 last 13.52) - open/close 6790 (time 1786308730.54 total 353.07 last 13.19) - open/close 6995 (time 1786308740.70 total 363.23 last 20.18) - open/close 7234 (time 1786308750.74 total 373.27 last 23.80) - open/close 7374 (time 1786308760.79 total 383.32 last 13.94) - open/close 7649 (time 1786308770.80 total 393.32 last 27.49) - open/close 7894 (time 1786308780.81 total 403.33 last 24.47) - open/close 8032 (time 1786308790.81 total 413.34 last 13.79) - open/close 8181 (time 1786308803.68 total 426.21 last 11.58) - open/close 8376 (time 1786308814.74 total 437.27 last 17.63) - open/close 8456 (time 1786308825.88 total 448.41 last 7.18) - open/close 8687 (time 1786308835.92 total 458.45 last 23.01) - open/close 8949 (time 1786308845.93 total 468.46 last 26.17) - open/close 9176 (time 1786308856.22 total 478.74 last 22.08) - open/close 9422 (time 1786308866.32 total 488.85 last 24.34) - open/close 9657 (time 1786308876.87 total 499.40 last 22.27) - open/close 9770 (time 1786308888.53 total 511.06 last 9.69) - open/close 9810 (time 1786308899.03 total 521.56 last 3.81) - open/close 9923 (time 1786308909.34 total 531.86 last 10.97) - open/close 10000 (time 1786308916.43 total 538.96 last 10.86) - open/close 10171 (time 1786308926.81 total 549.34 last 16.47) - open/close 10359 (time 1786308937.75 total 560.28 last 17.18) - open/close 10446 (time 1786308947.78 total 570.30 last 8.68) - open/close 10659 (time 1786308958.22 total 580.74 last 20.40) - open/close 10859 (time 1786308968.22 total 590.75 last 19.99) - open/close 11083 (time 1786308979.24 total 601.77 last 20.33) - open/close 11330 (time 1786308989.24 total 611.77 last 24.69) - open/close 11571 (time 1786308999.85 total 622.37 last 22.73) - open/close 11825 (time 1786309009.85 total 632.37 last 25.40) - open/close 12068 (time 1786309019.86 total 642.39 last 24.26) - open/close 12348 (time 1786309030.87 total 653.39 last 25.45) - open/close 12662 (time 1786309041.30 total 663.82 last 30.10) - open/close 12944 (time 1786309052.38 total 674.91 last 25.44) - open/close 13207 (time 1786309062.73 total 685.26 last 25.41) - open/close 13399 (time 1786309072.76 total 695.28 last 19.15) - open/close 13640 (time 1786309083.74 total 706.27 last 21.93) - open/close 13939 (time 1786309093.97 total 716.50 last 29.23) - open/close 14246 (time 1786309104.01 total 726.53 last 30.60) - open/close 14505 (time 1786309114.01 total 736.54 last 25.89) - open/close 14666 (time 1786309124.39 total 746.92 last 15.51) - open/close 14768 (time 1786309135.82 total 758.35 last 8.92) - open/close 14950 (time 1786309146.08 total 768.61 last 17.74) - open/close 15214 (time 1786309156.08 total 778.61 last 26.40) - open/close 15471 (time 1786309166.08 total 788.61 last 25.70) - open/close 15684 (time 1786309176.14 total 798.67 last 21.17) - open/close 15875 (time 1786309186.15 total 808.67 last 19.09) - open/close 16137 (time 1786309197.25 total 819.78 last 23.59) - open/close 16369 (time 1786309207.27 total 829.79 last 23.17) - open/close 16488 (time 1786309217.28 total 839.81 last 11.88) - open/close 16791 (time 1786309227.28 total 849.81 last 30.29) - open/close 16956 (time 1786309237.34 total 859.87 last 16.41) - open/close 17130 (time 1786309248.04 total 870.57 last 16.25) - open/close 17397 (time 1786309258.80 total 881.33 last 24.81) - open/close 17600 (time 1786309269.19 total 891.72 last 19.54) - open/close 17764 (time 1786309279.57 total 902.09 last 15.81) - open/close 17894 (time 1786309290.45 total 912.98 last 11.94) - open/close 18093 (time 1786309300.47 total 923.00 last 19.87) - open/close 18299 (time 1786309310.49 total 933.01 last 20.57) - open/close 18510 (time 1786309323.53 total 946.06 last 16.17) - open/close 18765 (time 1786309333.56 total 956.09 last 25.43) - open/close 18927 (time 1786309344.80 total 967.33 last 14.41) - open/close 19026 (time 1786309355.28 total 977.81 last 9.45) - open/close 19743 (time 1786309365.30 total 987.82 last 71.62) - open/close 20000 (time 1786309369.32 total 991.85 last 63.84) - open/close 20514 (time 1786309379.32 total 1001.85 last 51.40) - open/close 21102 (time 1786309389.32 total 1011.85 last 58.80) - open/close 21829 (time 1786309399.33 total 1021.86 last 72.62) - open/close 22575 (time 1786309409.33 total 1031.86 last 74.59) - open/close 23360 (time 1786309419.34 total 1041.86 last 78.47) - open/close 24131 (time 1786309429.36 total 1051.89 last 76.92) - open/close 24860 (time 1786309439.37 total 1061.90 last 72.80) - open/close 25630 (time 1786309449.38 total 1071.91 last 76.95) - open/close 26442 (time 1786309459.38 total 1081.91 last 81.18) - open/close 27217 (time 1786309469.40 total 1091.92 last 77.40) - open/close 27959 (time 1786309479.40 total 1101.93 last 74.17) - open/close 28737 (time 1786309489.41 total 1111.93 last 77.74) - open/close 29552 (time 1786309499.41 total 1121.94 last 81.47) - open/close 30000 (time 1786309504.92 total 1127.45 last 81.35) - open/close 30752 (time 1786309514.92 total 1137.45 last 75.19) - open/close 31542 (time 1786309524.92 total 1147.45 last 79.00) - open/close 32282 (time 1786309534.93 total 1157.46 last 73.94) - open/close 32984 (time 1786309544.94 total 1167.47 last 70.09) - open/close 33686 (time 1786309554.95 total 1177.48 last 70.16) - open/close 34520 (time 1786309564.96 total 1187.49 last 83.31) - open/close 35313 (time 1786309574.97 total 1197.50 last 79.24) - open/close 35968 (time 1786309584.98 total 1207.51 last 65.44) - open/close 36771 (time 1786309594.98 total 1217.51 last 80.26) - open/close 37515 (time 1786309604.99 total 1227.52 last 74.31) - open/close 38210 (time 1786309615.00 total 1237.53 last 69.44) - open/close 39016 (time 1786309625.00 total 1247.53 last 80.60) - open/close 39868 (time 1786309635.02 total 1257.54 last 85.08) - open/close 40679 (time 1786309645.04 total 1267.57 last 80.91) - open/close 41427 (time 1786309655.07 total 1277.60 last 74.57) - open/close 42132 (time 1786309665.08 total 1287.61 last 70.41) - open/close 42753 (time 1786309675.10 total 1297.63 last 62.00) - open/close 43398 (time 1786309685.11 total 1307.64 last 64.41) - open/close 44126 (time 1786309695.12 total 1317.64 last 72.78) - open/close 44903 (time 1786309705.12 total 1327.65 last 77.68) - open/close 45564 (time 1786309715.12 total 1337.65 last 66.09) - open/close 46303 (time 1786309725.13 total 1347.65 last 73.84) - open/close 47004 (time 1786309735.13 total 1357.66 last 70.07) - open/close 47762 (time 1786309745.14 total 1367.66 last 75.78) - open/close 48530 (time 1786309755.14 total 1377.67 last 76.76) - open/close 49322 (time 1786309765.15 total 1387.68 last 79.11) total: 50000 open/close in 1397.47 seconds: 35.78 ops/second - unlinked 0 (time 1786309779 ; total 0 ; last 0) - unlinked 10000 (time 1786309925 ; total 146 ; last 146) - unlinked 20000 (time 1786310058 ; total 279 ; last 133) - unlinked 30000 (time 1786310206 ; total 427 ; last 148) - unlinked 40000 (time 1786310337 ; total 558 ; last 131) total: 50000 unlinks in 686 seconds: 72.886299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2