== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:14:59 (1780996499) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 90 (time 1780996584.82 total 10.85 last 8.30) - open/close 238 (time 1780996595.78 total 21.81 last 13.50) - open/close 352 (time 1780996606.01 total 32.04 last 11.14) - open/close 481 (time 1780996616.44 total 42.48 last 12.36) - open/close 700 (time 1780996626.45 total 52.48 last 21.89) - open/close 838 (time 1780996636.46 total 62.49 last 13.79) - open/close 939 (time 1780996646.90 total 72.94 last 9.67) - open/close 1048 (time 1780996656.92 total 82.95 last 10.89) - open/close 1136 (time 1780996667.21 total 93.25 last 8.54) - open/close 1346 (time 1780996677.23 total 103.26 last 20.97) - open/close 1486 (time 1780996687.25 total 113.28 last 13.97) - open/close 1585 (time 1780996697.26 total 123.30 last 9.89) - open/close 1660 (time 1780996707.63 total 133.67 last 7.23) - open/close 1715 (time 1780996718.28 total 144.31 last 5.17) - open/close 1822 (time 1780996730.95 total 156.98 last 8.45) - open/close 1943 (time 1780996741.77 total 167.80 last 11.18) - open/close 2173 (time 1780996751.78 total 177.81 last 22.99) - open/close 2406 (time 1780996764.60 total 190.63 last 18.17) - open/close 2567 (time 1780996775.31 total 201.35 last 15.03) - open/close 2855 (time 1780996785.34 total 211.37 last 28.72) - open/close 2986 (time 1780996795.34 total 221.38 last 13.09) - open/close 3210 (time 1780996806.14 total 232.18 last 20.74) - open/close 3413 (time 1780996816.15 total 242.18 last 20.29) - open/close 3572 (time 1780996826.62 total 252.65 last 15.18) - open/close 3775 (time 1780996837.00 total 263.03 last 19.56) - open/close 3930 (time 1780996847.76 total 273.79 last 14.41) - open/close 4016 (time 1780996857.93 total 283.96 last 8.45) - open/close 4176 (time 1780996867.96 total 293.99 last 15.96) - open/close 4214 (time 1780996878.40 total 304.43 last 3.64) - open/close 4346 (time 1780996888.41 total 314.44 last 13.19) - open/close 4572 (time 1780996899.14 total 325.17 last 21.07) - open/close 4802 (time 1780996909.95 total 335.98 last 21.27) - open/close 5052 (time 1780996919.96 total 345.99 last 24.98) - open/close 5282 (time 1780996929.98 total 356.01 last 22.94) - open/close 5459 (time 1780996940.95 total 366.98 last 16.14) - open/close 5659 (time 1780996951.67 total 377.70 last 18.65) - open/close 5777 (time 1780996962.67 total 388.70 last 10.73) - open/close 5895 (time 1780996972.68 total 398.72 last 11.79) - open/close 6077 (time 1780996983.48 total 409.51 last 16.86) - open/close 6290 (time 1780996993.68 total 419.71 last 20.88) - open/close 6506 (time 1780997003.68 total 429.72 last 21.59) - open/close 6679 (time 1780997014.07 total 440.11 last 16.65) - open/close 6805 (time 1780997024.12 total 450.15 last 12.54) - open/close 6958 (time 1780997034.15 total 460.18 last 15.25) - open/close 7143 (time 1780997044.18 total 470.21 last 18.45) - open/close 7331 (time 1780997054.18 total 480.22 last 18.79) - open/close 7541 (time 1780997065.61 total 491.64 last 18.39) - open/close 7724 (time 1780997076.15 total 502.19 last 17.35) - open/close 7821 (time 1780997086.47 total 512.50 last 9.40) - open/close 7936 (time 1780997096.48 total 522.51 last 11.49) - open/close 8134 (time 1780997107.44 total 533.48 last 18.05) - open/close 8362 (time 1780997117.77 total 543.81 last 22.07) - open/close 8625 (time 1780997128.83 total 554.86 last 23.78) - open/close 8811 (time 1780997138.84 total 564.87 last 18.58) - open/close 9048 (time 1780997149.55 total 575.58 last 22.13) - open/close 9277 (time 1780997160.37 total 586.40 last 21.17) - open/close 9520 (time 1780997170.44 total 596.47 last 24.13) - open/close 9700 (time 1780997180.44 total 606.47 last 18.00) - open/close 9877 (time 1780997191.42 total 617.46 last 16.11) - open/close 10000 (time 1780997197.24 total 623.27 last 21.16) - open/close 10173 (time 1780997207.81 total 633.84 last 16.37) - open/close 10412 (time 1780997218.31 total 644.35 last 22.75) - open/close 10583 (time 1780997229.21 total 655.24 last 15.69) - open/close 10702 (time 1780997240.16 total 666.19 last 10.87) - open/close 10924 (time 1780997252.40 total 678.44 last 18.13) - open/close 11159 (time 1780997262.41 total 688.45 last 23.48) - open/close 11370 (time 1780997273.26 total 699.30 last 19.45) - open/close 11560 (time 1780997283.27 total 709.30 last 18.99) - open/close 11719 (time 1780997293.76 total 719.79 last 15.15) - open/close 11884 (time 1780997303.79 total 729.83 last 16.44) - open/close 11959 (time 1780997313.80 total 739.83 last 7.50) - open/close 12029 (time 1780997323.84 total 749.88 last 6.97) - open/close 12157 (time 1780997333.84 total 759.88 last 12.80) - open/close 12344 (time 1780997344.59 total 770.62 last 17.41) - open/close 12494 (time 1780997357.07 total 783.10 last 12.02) - open/close 12629 (time 1780997367.08 total 793.11 last 13.49) - open/close 12866 (time 1780997377.23 total 803.26 last 23.35) - open/close 13046 (time 1780997387.51 total 813.54 last 17.50) - open/close 13177 (time 1780997398.10 total 824.13 last 12.37) - open/close 13393 (time 1780997408.81 total 834.84 last 20.17) - open/close 13525 (time 1780997419.49 total 845.52 last 12.36) - open/close 13782 (time 1780997429.57 total 855.60 last 25.50) - open/close 14031 (time 1780997439.57 total 865.60 last 24.89) - open/close 14173 (time 1780997449.66 total 875.69 last 14.07) - open/close 14384 (time 1780997462.01 total 888.04 last 17.09) - open/close 14480 (time 1780997472.78 total 898.81 last 8.92) - open/close 14658 (time 1780997482.80 total 908.83 last 17.76) - open/close 14855 (time 1780997492.81 total 918.84 last 19.68) - open/close 15107 (time 1780997502.83 total 928.86 last 25.15) - open/close 15326 (time 1780997512.84 total 938.87 last 21.88) - open/close 15504 (time 1780997524.04 total 950.08 last 15.88) - open/close 15686 (time 1780997534.05 total 960.08 last 18.19) - open/close 15825 (time 1780997545.34 total 971.38 last 12.30) - open/close 16026 (time 1780997555.36 total 981.39 last 20.07) - open/close 16221 (time 1780997565.36 total 991.40 last 19.49) - open/close 16466 (time 1780997576.35 total 1002.38 last 22.30) - open/close 16655 (time 1780997587.38 total 1013.41 last 17.14) - open/close 16894 (time 1780997598.08 total 1024.11 last 22.33) - open/close 17582 (time 1780997608.08 total 1034.11 last 68.80) - open/close 18198 (time 1780997618.09 total 1044.12 last 61.56) - open/close 18861 (time 1780997628.10 total 1054.13 last 66.20) - open/close 19521 (time 1780997638.11 total 1064.14 last 65.98) - open/close 20000 (time 1780997645.35 total 1071.38 last 66.15) - open/close 20691 (time 1780997655.36 total 1081.39 last 69.03) - open/close 21344 (time 1780997665.37 total 1091.40 last 65.22) - open/close 21997 (time 1780997675.37 total 1101.40 last 65.30) - open/close 22639 (time 1780997685.38 total 1111.41 last 64.16) - open/close 23338 (time 1780997695.38 total 1121.41 last 69.90) - open/close 24006 (time 1780997705.40 total 1131.43 last 66.67) - open/close 24659 (time 1780997715.40 total 1141.43 last 65.28) - open/close 25310 (time 1780997725.41 total 1151.44 last 65.05) - open/close 26003 (time 1780997735.41 total 1161.44 last 69.26) - open/close 26687 (time 1780997745.42 total 1171.45 last 68.37) - open/close 27376 (time 1780997755.42 total 1181.46 last 68.84) - open/close 28030 (time 1780997765.45 total 1191.48 last 65.21) - open/close 28693 (time 1780997775.46 total 1201.49 last 66.28) - open/close 29365 (time 1780997785.46 total 1211.50 last 67.14) - open/close 29969 (time 1780997795.47 total 1221.50 last 60.39) - open/close 30645 (time 1780997805.47 total 1231.50 last 67.60) - open/close 31282 (time 1780997815.48 total 1241.51 last 63.60) - open/close 31924 (time 1780997825.48 total 1251.52 last 64.19) - open/close 32643 (time 1780997835.49 total 1261.52 last 71.84) - open/close 33290 (time 1780997845.50 total 1271.53 last 64.65) - open/close 33958 (time 1780997855.51 total 1281.54 last 66.75) - open/close 34646 (time 1780997865.51 total 1291.54 last 68.77) - open/close 35258 (time 1780997875.52 total 1301.55 last 61.16) - open/close 35872 (time 1780997885.53 total 1311.57 last 61.30) - open/close 36524 (time 1780997895.54 total 1321.57 last 65.16) - open/close 37177 (time 1780997905.56 total 1331.59 last 65.19) - open/close 37797 (time 1780997915.57 total 1341.61 last 61.89) - open/close 38376 (time 1780997925.58 total 1351.61 last 57.87) - open/close 38937 (time 1780997935.58 total 1361.61 last 56.08) - open/close 39579 (time 1780997945.58 total 1371.62 last 64.19) - open/close 40000 (time 1780997952.75 total 1378.78 last 58.76) - open/close 40646 (time 1780997962.76 total 1388.79 last 64.54) - open/close 41319 (time 1780997972.76 total 1398.79 last 67.30) - open/close 41962 (time 1780997982.76 total 1408.79 last 64.27) - open/close 42608 (time 1780997992.78 total 1418.81 last 64.51) - open/close 43263 (time 1780998002.79 total 1428.83 last 65.39) - open/close 43857 (time 1780998012.80 total 1438.83 last 59.36) - open/close 44492 (time 1780998022.81 total 1448.85 last 63.41) - open/close 45056 (time 1780998032.82 total 1458.85 last 56.39) - open/close 45721 (time 1780998042.84 total 1468.87 last 66.34) - open/close 46313 (time 1780998052.85 total 1478.88 last 59.12) - open/close 46965 (time 1780998062.85 total 1488.88 last 65.19) - open/close 47531 (time 1780998072.86 total 1498.89 last 56.54) - open/close 48079 (time 1780998082.87 total 1508.90 last 54.74) - open/close 48672 (time 1780998092.88 total 1518.92 last 59.24) - open/close 49326 (time 1780998102.89 total 1528.92 last 65.37) - open/close 49867 (time 1780998112.90 total 1538.93 last 54.02) total: 50000 open/close in 1541.70 seconds: 32.43 ops/second - unlinked 0 (time 1780998120 ; total 0 ; last 0) - unlinked 10000 (time 1780998291 ; total 171 ; last 171) - unlinked 20000 (time 1780998463 ; total 343 ; last 172) - unlinked 30000 (time 1780998636 ; total 516 ; last 173) - unlinked 40000 (time 1780998809 ; total 689 ; last 173) total: 50000 unlinks in 859 seconds: 58.207218 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2