== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:32:34 (1773610354) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 68 (time 1773610442.90 total 10.44 last 6.51) - open/close 226 (time 1773610454.41 total 21.95 last 13.73) - open/close 380 (time 1773610465.03 total 32.57 last 14.50) - open/close 577 (time 1773610475.04 total 42.58 last 19.68) - open/close 776 (time 1773610485.05 total 52.59 last 19.87) - open/close 1001 (time 1773610495.34 total 62.88 last 21.86) - open/close 1258 (time 1773610505.36 total 72.90 last 25.66) - open/close 1369 (time 1773610515.82 total 83.36 last 10.61) - open/close 1523 (time 1773610525.82 total 93.36 last 15.40) - open/close 1764 (time 1773610535.83 total 103.37 last 24.07) - open/close 1969 (time 1773610546.80 total 114.34 last 18.69) - open/close 2207 (time 1773610557.42 total 124.95 last 22.42) - open/close 2480 (time 1773610568.17 total 135.71 last 25.38) - open/close 2723 (time 1773610583.75 total 151.29 last 15.60) - open/close 2943 (time 1773610594.17 total 161.71 last 21.11) - open/close 3216 (time 1773610604.18 total 171.71 last 27.29) - open/close 3419 (time 1773610614.26 total 181.80 last 20.14) - open/close 3690 (time 1773610624.44 total 191.97 last 26.62) - open/close 3876 (time 1773610635.41 total 202.95 last 16.95) - open/close 4080 (time 1773610645.42 total 212.95 last 20.39) - open/close 4243 (time 1773610655.49 total 223.03 last 16.17) - open/close 4434 (time 1773610666.11 total 233.65 last 17.99) - open/close 4502 (time 1773610677.12 total 244.66 last 6.17) - open/close 4735 (time 1773610687.28 total 254.82 last 22.95) - open/close 4831 (time 1773610697.71 total 265.25 last 9.21) - open/close 4891 (time 1773610708.59 total 276.13 last 5.51) - open/close 5047 (time 1773610718.89 total 286.43 last 15.15) - open/close 5208 (time 1773610728.89 total 296.43 last 16.10) - open/close 5380 (time 1773610739.52 total 307.06 last 16.18) - open/close 5598 (time 1773610749.55 total 317.08 last 21.75) - open/close 5802 (time 1773610759.56 total 327.10 last 20.37) - open/close 6068 (time 1773610770.37 total 337.90 last 24.62) - open/close 6285 (time 1773610780.56 total 348.10 last 21.28) - open/close 6524 (time 1773610790.70 total 358.24 last 23.58) - open/close 6696 (time 1773610800.76 total 368.29 last 17.10) - open/close 6811 (time 1773610810.86 total 378.40 last 11.38) - open/close 7028 (time 1773610820.87 total 388.41 last 21.68) - open/close 7228 (time 1773610830.88 total 398.42 last 19.98) - open/close 7487 (time 1773610840.89 total 408.43 last 25.87) - open/close 7715 (time 1773610851.75 total 419.29 last 21.00) - open/close 7980 (time 1773610862.44 total 429.98 last 24.80) - open/close 8265 (time 1773610873.26 total 440.80 last 26.32) - open/close 8561 (time 1773610883.67 total 451.21 last 28.43) - open/close 8737 (time 1773610894.72 total 462.26 last 15.93) - open/close 8941 (time 1773610904.84 total 472.38 last 20.15) - open/close 9124 (time 1773610914.85 total 482.39 last 18.28) - open/close 9216 (time 1773610925.38 total 492.92 last 8.74) - open/close 9447 (time 1773610935.39 total 502.93 last 23.06) - open/close 9695 (time 1773610945.40 total 512.93 last 24.79) - open/close 9868 (time 1773610955.50 total 523.03 last 17.13) - open/close 10000 (time 1773610962.17 total 529.71 last 19.77) - open/close 10239 (time 1773610972.72 total 540.26 last 22.65) - open/close 10443 (time 1773610982.73 total 550.27 last 20.40) - open/close 10682 (time 1773610993.03 total 560.57 last 23.20) - open/close 10910 (time 1773611003.85 total 571.39 last 21.08) - open/close 11152 (time 1773611013.85 total 581.39 last 24.19) - open/close 11347 (time 1773611024.47 total 592.01 last 18.36) - open/close 11505 (time 1773611034.49 total 602.03 last 15.77) - open/close 11650 (time 1773611045.17 total 612.71 last 13.58) - open/close 11910 (time 1773611056.05 total 623.59 last 23.89) - open/close 12073 (time 1773611066.07 total 633.61 last 16.28) - open/close 12226 (time 1773611076.79 total 644.33 last 14.26) - open/close 12327 (time 1773611087.60 total 655.14 last 9.35) - open/close 12551 (time 1773611098.61 total 666.15 last 20.34) - open/close 12749 (time 1773611108.64 total 676.17 last 19.75) - open/close 12867 (time 1773611118.64 total 686.18 last 11.80) - open/close 12970 (time 1773611129.19 total 696.73 last 9.76) - open/close 13040 (time 1773611139.51 total 707.05 last 6.78) - open/close 13126 (time 1773611150.13 total 717.67 last 8.10) - open/close 13283 (time 1773611160.47 total 728.01 last 15.19) - open/close 13421 (time 1773611170.48 total 738.02 last 13.79) - open/close 13685 (time 1773611181.20 total 748.74 last 24.63) - open/close 13839 (time 1773611191.94 total 759.48 last 14.34) - open/close 14024 (time 1773611201.94 total 769.48 last 18.49) - open/close 14293 (time 1773611211.95 total 779.49 last 26.88) - open/close 14533 (time 1773611221.96 total 789.50 last 23.98) - open/close 14683 (time 1773611231.97 total 799.51 last 14.98) - open/close 14875 (time 1773611241.97 total 809.51 last 19.20) - open/close 15037 (time 1773611252.50 total 820.04 last 15.39) - open/close 15254 (time 1773611262.84 total 830.38 last 21.00) - open/close 15454 (time 1773611274.24 total 841.78 last 17.54) - open/close 15581 (time 1773611284.25 total 851.79 last 12.69) - open/close 15837 (time 1773611294.26 total 861.80 last 25.57) - open/close 16108 (time 1773611304.27 total 871.81 last 27.09) - open/close 16389 (time 1773611314.29 total 881.83 last 28.04) - open/close 16636 (time 1773611324.98 total 892.52 last 23.10) - open/close 16868 (time 1773611335.82 total 903.36 last 21.41) - open/close 16954 (time 1773611346.75 total 914.29 last 7.86) - open/close 17148 (time 1773611357.24 total 924.77 last 18.51) - open/close 17316 (time 1773611367.88 total 935.42 last 15.78) - open/close 17566 (time 1773611378.96 total 946.49 last 22.57) - open/close 17779 (time 1773611389.57 total 957.11 last 20.07) - open/close 17929 (time 1773611399.90 total 967.44 last 14.52) - open/close 18180 (time 1773611409.91 total 977.45 last 25.07) - open/close 18357 (time 1773611420.41 total 987.95 last 16.86) - open/close 18623 (time 1773611430.42 total 997.96 last 26.57) - open/close 19466 (time 1773611440.43 total 1007.97 last 84.21) - open/close 20000 (time 1773611447.08 total 1014.62 last 80.38) - open/close 20644 (time 1773611457.08 total 1024.62 last 64.35) - open/close 21381 (time 1773611467.11 total 1034.65 last 73.49) - open/close 22096 (time 1773611477.12 total 1044.66 last 71.46) - open/close 22849 (time 1773611487.13 total 1054.67 last 75.23) - open/close 23659 (time 1773611497.13 total 1064.67 last 80.99) - open/close 24344 (time 1773611507.14 total 1074.68 last 68.42) - open/close 25013 (time 1773611517.15 total 1084.69 last 66.83) - open/close 25727 (time 1773611527.16 total 1094.70 last 71.30) - open/close 26453 (time 1773611537.17 total 1104.71 last 72.54) - open/close 27100 (time 1773611547.18 total 1114.72 last 64.66) - open/close 27855 (time 1773611557.18 total 1124.72 last 75.50) - open/close 28554 (time 1773611567.18 total 1134.72 last 69.86) - open/close 29334 (time 1773611577.19 total 1144.72 last 78.00) - open/close 30000 (time 1773611585.93 total 1153.47 last 76.15) - open/close 30682 (time 1773611595.94 total 1163.48 last 68.11) - open/close 31533 (time 1773611605.96 total 1173.50 last 84.97) - open/close 32385 (time 1773611615.96 total 1183.50 last 85.16) - open/close 33299 (time 1773611625.96 total 1193.50 last 91.40) - open/close 34180 (time 1773611635.97 total 1203.51 last 88.05) - open/close 35037 (time 1773611645.98 total 1213.52 last 85.58) - open/close 35881 (time 1773611655.99 total 1223.53 last 84.35) - open/close 36566 (time 1773611666.04 total 1233.58 last 68.16) - open/close 37391 (time 1773611676.05 total 1243.59 last 82.44) - open/close 38240 (time 1773611686.06 total 1253.59 last 84.84) - open/close 38985 (time 1773611696.06 total 1263.60 last 74.47) - open/close 39854 (time 1773611706.06 total 1273.60 last 86.89) - open/close 40623 (time 1773611716.07 total 1283.61 last 76.84) - open/close 41552 (time 1773611726.08 total 1293.62 last 92.82) - open/close 42414 (time 1773611736.09 total 1303.63 last 86.11) - open/close 43075 (time 1773611746.09 total 1313.63 last 66.09) - open/close 43933 (time 1773611756.09 total 1323.63 last 85.79) - open/close 44779 (time 1773611766.10 total 1333.64 last 84.49) - open/close 45694 (time 1773611776.10 total 1343.64 last 91.49) - open/close 46617 (time 1773611786.11 total 1353.65 last 92.24) - open/close 47543 (time 1773611796.11 total 1363.65 last 92.56) - open/close 48442 (time 1773611806.11 total 1373.65 last 89.89) - open/close 49257 (time 1773611816.12 total 1383.65 last 81.48) total: 50000 open/close in 1393.29 seconds: 35.89 ops/second - unlinked 0 (time 1773611829 ; total 0 ; last 0) - unlinked 10000 (time 1773611953 ; total 124 ; last 124) - unlinked 20000 (time 1773612068 ; total 239 ; last 115) - unlinked 30000 (time 1773612171 ; total 342 ; last 103) - unlinked 40000 (time 1773612245 ; total 416 ; last 74) total: 50000 unlinks in 483 seconds: 103.519669 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2