== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:14:03 (1788167643) start mds service on oleg248-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg248-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg248-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1788167729.41 total 10.25 last 21.08) - open/close 450 (time 1788167739.91 total 20.75 last 22.28) - open/close 710 (time 1788167750.54 total 31.39 last 24.45) - open/close 957 (time 1788167760.55 total 41.39 last 24.68) - open/close 1167 (time 1788167770.58 total 51.43 last 20.93) - open/close 1324 (time 1788167780.60 total 61.44 last 15.68) - open/close 1515 (time 1788167790.62 total 71.46 last 19.06) - open/close 1671 (time 1788167800.63 total 81.47 last 15.58) - open/close 1882 (time 1788167811.08 total 91.92 last 20.19) - open/close 2180 (time 1788167822.09 total 102.93 last 27.07) - open/close 2451 (time 1788167832.83 total 113.68 last 25.23) - open/close 2748 (time 1788167842.85 total 123.69 last 29.66) - open/close 2994 (time 1788167853.82 total 134.67 last 22.41) - open/close 3277 (time 1788167863.84 total 144.68 last 28.27) - open/close 3494 (time 1788167873.84 total 154.68 last 21.69) - open/close 3697 (time 1788167883.84 total 164.68 last 20.30) - open/close 3920 (time 1788167894.09 total 174.93 last 21.77) - open/close 4195 (time 1788167905.02 total 185.86 last 25.15) - open/close 4438 (time 1788167915.02 total 195.86 last 24.29) - open/close 4678 (time 1788167926.03 total 206.87 last 21.80) - open/close 4920 (time 1788167937.04 total 217.88 last 21.99) - open/close 5093 (time 1788167947.79 total 228.63 last 16.09) - open/close 5378 (time 1788167957.81 total 238.65 last 28.45) - open/close 5663 (time 1788167968.77 total 249.61 last 26.00) - open/close 5890 (time 1788167978.79 total 259.63 last 22.66) - open/close 6156 (time 1788167988.81 total 269.65 last 26.53) - open/close 6378 (time 1788167999.38 total 280.22 last 21.01) - open/close 6598 (time 1788168009.38 total 290.22 last 22.00) - open/close 6816 (time 1788168019.40 total 300.24 last 21.76) - open/close 7056 (time 1788168029.42 total 310.26 last 23.95) - open/close 7235 (time 1788168039.44 total 320.28 last 17.86) - open/close 7367 (time 1788168049.45 total 330.29 last 13.18) - open/close 7435 (time 1788168059.49 total 340.33 last 6.77) - open/close 7543 (time 1788168070.04 total 350.88 last 10.24) - open/close 7752 (time 1788168080.46 total 361.30 last 20.06) - open/close 8011 (time 1788168090.48 total 371.32 last 25.84) - open/close 8246 (time 1788168100.94 total 381.78 last 22.48) - open/close 8472 (time 1788168111.51 total 392.35 last 21.37) - open/close 8755 (time 1788168121.52 total 402.36 last 28.28) - open/close 8977 (time 1788168131.54 total 412.38 last 22.15) - open/close 9231 (time 1788168141.90 total 422.74 last 24.52) - open/close 9483 (time 1788168152.68 total 433.52 last 23.38) - open/close 9687 (time 1788168163.27 total 444.11 last 19.27) - open/close 9881 (time 1788168173.32 total 454.16 last 19.30) - open/close 10000 (time 1788168178.47 total 459.31 last 23.08) - open/close 10117 (time 1788168189.15 total 469.99 last 10.95) - open/close 10383 (time 1788168200.13 total 480.97 last 24.23) - open/close 10621 (time 1788168210.88 total 491.72 last 22.13) - open/close 10801 (time 1788168220.91 total 501.75 last 17.95) - open/close 10920 (time 1788168230.91 total 511.75 last 11.90) - open/close 11035 (time 1788168240.91 total 521.76 last 11.50) - open/close 11180 (time 1788168250.92 total 531.76 last 14.49) - open/close 11316 (time 1788168261.37 total 542.21 last 13.02) - open/close 11457 (time 1788168271.39 total 552.23 last 14.07) - open/close 11607 (time 1788168281.56 total 562.40 last 14.74) - open/close 11772 (time 1788168291.61 total 572.45 last 16.43) - open/close 11912 (time 1788168307.37 total 588.21 last 8.88) - open/close 12037 (time 1788168317.52 total 598.36 last 12.31) - open/close 12157 (time 1788168328.39 total 609.24 last 11.04) - open/close 12295 (time 1788168338.42 total 619.26 last 13.77) - open/close 12469 (time 1788168349.06 total 629.90 last 16.35) - open/close 12626 (time 1788168359.79 total 640.63 last 14.63) - open/close 12757 (time 1788168370.59 total 651.43 last 12.13) - open/close 12995 (time 1788168381.56 total 662.41 last 21.69) - open/close 13256 (time 1788168392.28 total 673.12 last 24.35) - open/close 13534 (time 1788168402.30 total 683.14 last 27.75) - open/close 13579 (time 1788168412.68 total 693.53 last 4.33) - open/close 13783 (time 1788168422.71 total 703.56 last 20.34) - open/close 13898 (time 1788168435.34 total 716.18 last 9.11) - open/close 14145 (time 1788168445.34 total 726.19 last 24.69) - open/close 14298 (time 1788168455.63 total 736.47 last 14.87) - open/close 14560 (time 1788168466.25 total 747.09 last 24.68) - open/close 14813 (time 1788168476.62 total 757.47 last 24.38) - open/close 15090 (time 1788168487.36 total 768.20 last 25.80) - open/close 15366 (time 1788168497.37 total 778.21 last 27.58) - open/close 15637 (time 1788168507.37 total 788.21 last 27.09) - open/close 15758 (time 1788168517.38 total 798.22 last 12.09) - open/close 15985 (time 1788168527.81 total 808.65 last 21.77) - open/close 16144 (time 1788168538.26 total 819.11 last 15.20) - open/close 16436 (time 1788168548.45 total 829.29 last 28.68) - open/close 16634 (time 1788168558.46 total 839.30 last 19.78) - open/close 16825 (time 1788168568.46 total 849.30 last 19.09) - open/close 16977 (time 1788168579.95 total 860.79 last 13.23) - open/close 17082 (time 1788168592.72 total 873.56 last 8.22) - open/close 17267 (time 1788168602.81 total 883.65 last 18.34) - open/close 17460 (time 1788168615.50 total 896.35 last 15.21) - open/close 17681 (time 1788168625.50 total 906.35 last 22.10) - open/close 17869 (time 1788168636.17 total 917.02 last 17.62) - open/close 18041 (time 1788168646.85 total 927.69 last 16.11) - open/close 18312 (time 1788168656.86 total 937.70 last 27.08) - open/close 18476 (time 1788168669.89 total 950.73 last 12.58) - open/close 18779 (time 1788168679.90 total 960.74 last 30.28) - open/close 19496 (time 1788168689.91 total 970.75 last 71.63) - open/close 20000 (time 1788168697.67 total 978.51 last 64.92) - open/close 20779 (time 1788168707.67 total 988.52 last 77.88) - open/close 21548 (time 1788168717.68 total 998.52 last 76.85) - open/close 22236 (time 1788168727.69 total 1008.53 last 68.74) - open/close 22894 (time 1788168737.70 total 1018.54 last 65.73) - open/close 23542 (time 1788168747.71 total 1028.55 last 64.73) - open/close 24247 (time 1788168757.72 total 1038.56 last 70.43) - open/close 24990 (time 1788168767.73 total 1048.58 last 74.21) - open/close 25636 (time 1788168777.74 total 1058.58 last 64.56) - open/close 26384 (time 1788168787.76 total 1068.60 last 74.63) - open/close 27126 (time 1788168797.81 total 1078.66 last 73.81) - open/close 27769 (time 1788168807.84 total 1088.68 last 64.15) - open/close 28471 (time 1788168817.85 total 1098.69 last 70.10) - open/close 29146 (time 1788168827.86 total 1108.70 last 67.44) - open/close 29751 (time 1788168837.87 total 1118.71 last 60.44) - open/close 30000 (time 1788168842.26 total 1123.10 last 56.73) - open/close 30715 (time 1788168852.28 total 1133.12 last 71.38) - open/close 31475 (time 1788168862.29 total 1143.13 last 75.94) - open/close 32124 (time 1788168872.29 total 1153.13 last 64.90) - open/close 32805 (time 1788168882.29 total 1163.13 last 68.10) - open/close 33578 (time 1788168892.29 total 1173.14 last 77.26) - open/close 34306 (time 1788168902.29 total 1183.14 last 72.79) - open/close 34977 (time 1788168912.30 total 1193.14 last 67.07) - open/close 35601 (time 1788168922.31 total 1203.15 last 62.32) - open/close 36305 (time 1788168932.32 total 1213.16 last 70.34) - open/close 36987 (time 1788168942.32 total 1223.17 last 68.16) - open/close 37678 (time 1788168952.33 total 1233.17 last 69.07) - open/close 38390 (time 1788168962.33 total 1243.18 last 71.16) - open/close 39121 (time 1788168972.35 total 1253.19 last 72.99) - open/close 39781 (time 1788168982.36 total 1263.20 last 65.96) - open/close 40000 (time 1788168985.87 total 1266.71 last 62.34) - open/close 40612 (time 1788168995.89 total 1276.73 last 61.07) - open/close 41139 (time 1788169005.90 total 1286.74 last 52.65) - open/close 41790 (time 1788169015.90 total 1296.75 last 65.07) - open/close 42452 (time 1788169025.91 total 1306.75 last 66.16) - open/close 43180 (time 1788169035.92 total 1316.76 last 72.73) - open/close 43906 (time 1788169045.94 total 1326.78 last 72.48) - open/close 44643 (time 1788169055.95 total 1336.79 last 73.61) - open/close 45277 (time 1788169065.98 total 1346.82 last 63.23) - open/close 45807 (time 1788169075.98 total 1356.82 last 52.99) - open/close 46314 (time 1788169086.01 total 1366.85 last 50.54) - open/close 46902 (time 1788169096.02 total 1376.86 last 58.76) - open/close 47479 (time 1788169106.02 total 1386.86 last 57.67) - open/close 48135 (time 1788169116.02 total 1396.87 last 65.57) - open/close 48815 (time 1788169126.04 total 1406.88 last 67.92) - open/close 49492 (time 1788169136.06 total 1416.90 last 67.57) total: 50000 open/close in 1424.34 seconds: 35.10 ops/second - unlinked 0 (time 1788169148 ; total 0 ; last 0) - unlinked 10000 (time 1788169322 ; total 174 ; last 174) - unlinked 20000 (time 1788169491 ; total 343 ; last 169) - unlinked 30000 (time 1788169654 ; total 506 ; last 163) - unlinked 40000 (time 1788169814 ; total 666 ; last 160) total: 50000 unlinks in 835 seconds: 59.880241 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2