== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:10:51 (1787389851) start mds service on oleg632-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg632-server' oleg632-server: oleg632-server.virtnet: executing load_modules_local oleg632-server: Loading modules from /home/green/git/lustre-release/lustre oleg632-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg632-server: detected 4 online CPUs by sysfs oleg632-server: MODOPTS_LIBCFS= oleg632-server: Force libcfs to create 2 CPU partitions oleg632-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg632-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg632-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg632-server: oleg632-server.virtnet: executing set_default_debug -1 all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg632-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug -1 all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 212 (time 1787389919.88 total 10.85 last 19.54) - open/close 444 (time 1787389929.97 total 20.95 last 22.98) - open/close 720 (time 1787389940.50 total 31.48 last 26.22) - open/close 989 (time 1787389950.50 total 41.48 last 26.89) - open/close 1278 (time 1787389960.82 total 51.80 last 28.01) - open/close 1490 (time 1787389971.60 total 62.57 last 19.67) - open/close 1694 (time 1787389982.02 total 73.00 last 19.57) - open/close 1943 (time 1787389992.96 total 83.93 last 22.77) - open/close 2135 (time 1787390002.97 total 93.94 last 19.19) - open/close 2376 (time 1787390013.63 total 104.60 last 22.60) - open/close 2619 (time 1787390024.32 total 115.29 last 22.74) - open/close 2882 (time 1787390034.33 total 125.30 last 26.28) - open/close 3117 (time 1787390044.34 total 135.31 last 23.47) - open/close 3354 (time 1787390054.35 total 145.33 last 23.66) - open/close 3595 (time 1787390064.58 total 155.55 last 23.57) - open/close 3832 (time 1787390074.68 total 165.65 last 23.47) - open/close 4085 (time 1787390084.69 total 175.67 last 25.26) - open/close 4292 (time 1787390095.50 total 186.48 last 19.15) - open/close 4495 (time 1787390105.51 total 196.49 last 20.28) - open/close 4717 (time 1787390115.51 total 206.49 last 22.20) - open/close 4917 (time 1787390125.53 total 216.50 last 19.97) - open/close 5154 (time 1787390135.99 total 226.97 last 22.65) - open/close 5418 (time 1787390146.42 total 237.40 last 25.31) - open/close 5662 (time 1787390156.67 total 247.64 last 23.81) - open/close 5928 (time 1787390166.86 total 257.83 last 26.11) - open/close 6203 (time 1787390176.86 total 267.84 last 27.49) - open/close 6471 (time 1787390187.25 total 278.23 last 25.79) - open/close 6752 (time 1787390198.28 total 289.25 last 25.49) - open/close 7009 (time 1787390208.64 total 299.61 last 24.80) - open/close 7195 (time 1787390219.68 total 310.65 last 16.85) - open/close 7460 (time 1787390230.22 total 321.19 last 25.14) - open/close 7715 (time 1787390240.86 total 331.83 last 23.97) - open/close 7997 (time 1787390250.87 total 341.85 last 28.15) - open/close 8205 (time 1787390261.82 total 352.80 last 19.00) - open/close 8418 (time 1787390271.83 total 362.81 last 21.28) - open/close 8662 (time 1787390282.43 total 373.40 last 23.03) - open/close 8917 (time 1787390292.46 total 383.43 last 25.43) - open/close 9091 (time 1787390302.79 total 393.76 last 16.84) - open/close 9322 (time 1787390312.84 total 403.81 last 22.99) - open/close 9555 (time 1787390323.11 total 414.08 last 22.69) - open/close 9809 (time 1787390333.47 total 424.44 last 24.51) - open/close 10000 (time 1787390340.93 total 431.90 last 25.61) - open/close 10235 (time 1787390350.93 total 441.90 last 23.49) - open/close 10477 (time 1787390360.95 total 451.92 last 24.16) - open/close 10730 (time 1787390371.59 total 462.56 last 23.77) - open/close 10996 (time 1787390381.61 total 472.58 last 26.55) - open/close 11231 (time 1787390391.93 total 482.90 last 22.77) - open/close 11478 (time 1787390402.24 total 493.21 last 23.96) - open/close 11664 (time 1787390412.25 total 503.23 last 18.57) - open/close 11909 (time 1787390422.66 total 513.63 last 23.55) - open/close 12181 (time 1787390432.67 total 523.64 last 27.17) - open/close 12384 (time 1787390442.68 total 533.66 last 20.27) - open/close 12547 (time 1787390453.18 total 544.16 last 15.53) - open/close 12726 (time 1787390464.27 total 555.24 last 16.15) - open/close 12988 (time 1787390474.27 total 565.24 last 26.20) - open/close 13239 (time 1787390484.28 total 575.25 last 25.08) - open/close 13448 (time 1787390495.18 total 586.16 last 19.16) - open/close 13683 (time 1787390505.82 total 596.79 last 22.10) - open/close 13899 (time 1787390515.82 total 606.80 last 21.59) - open/close 14156 (time 1787390526.14 total 617.11 last 24.92) - open/close 14405 (time 1787390537.02 total 628.00 last 22.87) - open/close 14661 (time 1787390547.92 total 638.89 last 23.49) - open/close 14918 (time 1787390558.42 total 649.39 last 24.48) - open/close 15201 (time 1787390569.35 total 660.32 last 25.90) - open/close 15454 (time 1787390580.37 total 671.34 last 22.96) - open/close 15714 (time 1787390590.84 total 681.81 last 24.82) - open/close 15978 (time 1787390600.90 total 691.87 last 26.24) - open/close 16239 (time 1787390610.91 total 701.88 last 26.09) - open/close 16504 (time 1787390621.82 total 712.79 last 24.29) - open/close 16753 (time 1787390632.02 total 722.99 last 24.41) - open/close 17024 (time 1787390642.90 total 733.87 last 24.91) - open/close 17160 (time 1787390653.52 total 744.49 last 12.81) - open/close 17311 (time 1787390663.81 total 754.78 last 14.67) - open/close 17500 (time 1787390675.46 total 766.43 last 16.22) - open/close 17761 (time 1787390686.34 total 777.31 last 23.99) - open/close 17942 (time 1787390696.35 total 787.33 last 18.08) - open/close 18244 (time 1787390706.37 total 797.34 last 30.15) - open/close 18506 (time 1787390716.75 total 807.72 last 25.25) - open/close 18729 (time 1787390727.49 total 818.46 last 20.76) - open/close 18966 (time 1787390737.98 total 828.95 last 22.59) - open/close 19203 (time 1787390747.99 total 838.96 last 23.67) - open/close 19460 (time 1787390758.91 total 849.88 last 23.53) - open/close 19704 (time 1787390769.30 total 860.28 last 23.48) - open/close 19971 (time 1787390779.31 total 870.28 last 26.68) - open/close 20690 (time 1787390789.32 total 880.30 last 71.83) - open/close 21556 (time 1787390799.34 total 890.32 last 86.43) - open/close 22455 (time 1787390809.35 total 900.32 last 89.84) - open/close 23363 (time 1787390819.35 total 910.32 last 90.80) - open/close 24267 (time 1787390829.35 total 920.33 last 90.36) - open/close 25176 (time 1787390839.35 total 930.33 last 90.89) - open/close 26114 (time 1787390849.38 total 940.36 last 93.52) - open/close 26974 (time 1787390859.39 total 950.36 last 85.97) - open/close 27858 (time 1787390869.39 total 960.36 last 88.39) - open/close 28731 (time 1787390879.39 total 970.37 last 87.25) - open/close 29590 (time 1787390889.40 total 980.38 last 85.81) - open/close 30000 (time 1787390894.22 total 985.19 last 85.14) - open/close 30826 (time 1787390904.23 total 995.20 last 82.52) - open/close 31661 (time 1787390914.23 total 1005.21 last 83.48) - open/close 32488 (time 1787390924.24 total 1015.21 last 82.65) - open/close 33381 (time 1787390934.24 total 1025.21 last 89.29) - open/close 34309 (time 1787390944.24 total 1035.22 last 92.77) - open/close 35159 (time 1787390954.25 total 1045.23 last 84.92) - open/close 36019 (time 1787390964.26 total 1055.23 last 85.95) - open/close 36843 (time 1787390974.26 total 1065.23 last 82.39) - open/close 37666 (time 1787390984.27 total 1075.24 last 82.26) - open/close 38535 (time 1787390994.27 total 1085.24 last 86.88) - open/close 39303 (time 1787391004.29 total 1095.26 last 76.64) - open/close 40000 (time 1787391012.72 total 1103.69 last 82.71) - open/close 40835 (time 1787391022.72 total 1113.70 last 83.43) - open/close 41612 (time 1787391032.74 total 1123.71 last 77.61) - open/close 42478 (time 1787391042.74 total 1133.71 last 86.58) - open/close 43082 (time 1787391052.74 total 1143.72 last 60.38) - open/close 43777 (time 1787391062.75 total 1153.72 last 69.47) - open/close 44583 (time 1787391072.75 total 1163.72 last 80.56) - open/close 45396 (time 1787391082.76 total 1173.73 last 81.24) - open/close 46181 (time 1787391092.76 total 1183.73 last 78.48) - open/close 46945 (time 1787391102.76 total 1193.74 last 76.38) - open/close 47798 (time 1787391112.77 total 1203.74 last 85.24) - open/close 48573 (time 1787391122.78 total 1213.75 last 77.43) - open/close 49352 (time 1787391132.78 total 1223.75 last 77.90) total: 50000 open/close in 1230.76 seconds: 40.63 ops/second - unlinked 0 (time 1787391144 ; total 0 ; last 0) - unlinked 10000 (time 1787391271 ; total 127 ; last 127) - unlinked 20000 (time 1787391407 ; total 263 ; last 136) - unlinked 30000 (time 1787391542 ; total 398 ; last 135) - unlinked 40000 (time 1787391679 ; total 535 ; last 137) total: 50000 unlinks in 671 seconds: 74.515648 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg632-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg632-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg632-server stop mds service on oleg632-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg632-server unloading modules via unload_modules_local on: 'oleg632-server' oleg632-server: oleg632-server.virtnet: executing unload_modules_local oleg632-server: modules unloaded. pdsh@oleg632-client: oleg632-client: ssh exited with exit code 2 pdsh@oleg632-client: oleg632-server: ssh exited with exit code 2 pdsh@oleg632-client: oleg632-server: ssh exited with exit code 2 pdsh@oleg632-client: oleg632-client: ssh exited with exit code 2