== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:23:20 (1785936200) start mds service on oleg224-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 422 (time 1785936318.69 total 11.84 last 35.65) - open/close 608 (time 1785936328.70 total 21.85 last 18.58) - open/close 810 (time 1785936338.81 total 31.96 last 19.97) - open/close 930 (time 1785936348.99 total 42.14 last 11.80) - open/close 1046 (time 1785936358.99 total 52.14 last 11.59) - open/close 1215 (time 1785936369.02 total 62.17 last 16.86) - open/close 1401 (time 1785936379.85 total 73.00 last 17.18) - open/close 1525 (time 1785936389.85 total 83.00 last 12.40) - open/close 1796 (time 1785936399.88 total 93.03 last 27.03) - open/close 2009 (time 1785936410.14 total 103.29 last 20.76) - open/close 2152 (time 1785936420.15 total 113.30 last 14.28) - open/close 2247 (time 1785936430.46 total 123.61 last 9.21) - open/close 2356 (time 1785936440.51 total 133.66 last 10.85) - open/close 2514 (time 1785936450.52 total 143.67 last 15.79) - open/close 2690 (time 1785936461.14 total 154.29 last 16.57) - open/close 2880 (time 1785936471.15 total 164.30 last 18.98) - open/close 3032 (time 1785936481.56 total 174.71 last 14.60) - open/close 3141 (time 1785936492.36 total 185.51 last 10.10) - open/close 3319 (time 1785936502.38 total 195.53 last 17.76) - open/close 3508 (time 1785936512.40 total 205.55 last 18.87) - open/close 3695 (time 1785936522.42 total 215.57 last 18.65) - open/close 3860 (time 1785936532.72 total 225.87 last 16.02) - open/close 3938 (time 1785936542.83 total 235.98 last 7.72) - open/close 4055 (time 1785936553.28 total 246.43 last 11.19) - open/close 4145 (time 1785936563.84 total 256.99 last 8.52) - open/close 4209 (time 1785936573.85 total 267.00 last 6.40) - open/close 4331 (time 1785936583.86 total 277.01 last 12.19) - open/close 4471 (time 1785936596.61 total 289.76 last 10.98) - open/close 4573 (time 1785936606.89 total 300.04 last 9.92) - open/close 4665 (time 1785936616.91 total 310.06 last 9.18) - open/close 4748 (time 1785936627.64 total 320.79 last 7.73) - open/close 4868 (time 1785936637.65 total 330.80 last 12.00) - open/close 5005 (time 1785936647.69 total 340.84 last 13.63) - open/close 5175 (time 1785936658.42 total 351.57 last 15.86) - open/close 5370 (time 1785936668.83 total 361.98 last 18.72) - open/close 5496 (time 1785936679.18 total 372.33 last 12.17) - open/close 5697 (time 1785936689.20 total 382.35 last 20.08) - open/close 5773 (time 1785936699.64 total 392.79 last 7.28) - open/close 5869 (time 1785936710.53 total 403.68 last 8.82) - open/close 6011 (time 1785936720.78 total 413.93 last 13.86) - open/close 6107 (time 1785936731.63 total 424.78 last 8.84) - open/close 6158 (time 1785936742.41 total 435.56 last 4.73) - open/close 6288 (time 1785936752.49 total 445.64 last 12.90) - open/close 6405 (time 1785936762.78 total 455.92 last 11.38) - open/close 6525 (time 1785936772.82 total 465.97 last 11.95) - open/close 6637 (time 1785936782.84 total 475.99 last 11.17) - open/close 6797 (time 1785936792.86 total 486.01 last 15.97) - open/close 6897 (time 1785936804.27 total 497.42 last 8.77) - open/close 7004 (time 1785936814.27 total 507.42 last 10.70) - open/close 7136 (time 1785936824.63 total 517.78 last 12.75) - open/close 7219 (time 1785936834.93 total 528.08 last 8.05) - open/close 7275 (time 1785936844.94 total 538.09 last 5.60) - open/close 7400 (time 1785936855.65 total 548.80 last 11.67) - open/close 7516 (time 1785936866.50 total 559.65 last 10.68) - open/close 7693 (time 1785936876.53 total 569.68 last 17.66) - open/close 7796 (time 1785936887.38 total 580.53 last 9.49) - open/close 7979 (time 1785936897.39 total 590.54 last 18.29) - open/close 8143 (time 1785936907.85 total 601.00 last 15.67) - open/close 8264 (time 1785936917.91 total 611.06 last 12.03) - open/close 8394 (time 1785936928.41 total 621.56 last 12.39) - open/close 8446 (time 1785936939.53 total 632.68 last 4.68) - open/close 8650 (time 1785936950.58 total 643.73 last 18.46) - open/close 8768 (time 1785936960.83 total 653.98 last 11.52) - open/close 8882 (time 1785936971.03 total 664.18 last 11.18) - open/close 9088 (time 1785936981.04 total 674.19 last 20.57) - open/close 9237 (time 1785936991.08 total 684.23 last 14.85) - open/close 9344 (time 1785937002.06 total 695.21 last 9.74) - open/close 9511 (time 1785937012.07 total 705.22 last 16.70) - open/close 9718 (time 1785937022.35 total 715.50 last 20.13) - open/close 9897 (time 1785937032.36 total 725.51 last 17.88) - open/close 10000 (time 1785937041.12 total 734.27 last 11.76) - open/close 10105 (time 1785937051.84 total 744.99 last 9.80) - open/close 10221 (time 1785937061.90 total 755.05 last 11.54) - open/close 10357 (time 1785937072.89 total 766.04 last 12.37) - open/close 10465 (time 1785937083.90 total 777.05 last 9.81) - open/close 10551 (time 1785937094.52 total 787.67 last 8.09) - open/close 10608 (time 1785937104.54 total 797.69 last 5.69) - open/close 10732 (time 1785937114.65 total 807.80 last 12.27) - open/close 10842 (time 1785937124.65 total 817.80 last 10.99) - open/close 10929 (time 1785937135.57 total 828.72 last 7.97) - open/close 11094 (time 1785937145.60 total 838.75 last 16.44) - open/close 11252 (time 1785937155.62 total 848.77 last 15.77) - open/close 11380 (time 1785937169.21 total 862.35 last 9.43) - open/close 11612 (time 1785937179.22 total 872.37 last 23.17) - open/close 11776 (time 1785937189.84 total 882.99 last 15.44) - open/close 11839 (time 1785937200.14 total 893.29 last 6.11) - open/close 11946 (time 1785937210.44 total 903.59 last 10.39) - open/close 12020 (time 1785937220.81 total 913.96 last 7.13) - open/close 12134 (time 1785937230.91 total 924.06 last 11.28) - open/close 12215 (time 1785937241.26 total 934.41 last 7.83) - open/close 12384 (time 1785937251.28 total 944.43 last 16.88) - open/close 12491 (time 1785937261.82 total 954.97 last 10.15) - open/close 12557 (time 1785937274.01 total 967.16 last 5.41) - open/close 12696 (time 1785937284.41 total 977.55 last 13.38) - open/close 12825 (time 1785937294.86 total 988.01 last 12.34) - open/close 12995 (time 1785937304.95 total 998.10 last 16.85) - open/close 13140 (time 1785937315.97 total 1009.12 last 13.16) - open/close 13349 (time 1785937326.60 total 1019.74 last 19.67) - open/close 13447 (time 1785937337.61 total 1030.76 last 8.90) - open/close 13524 (time 1785937348.10 total 1041.25 last 7.34) - open/close 13731 (time 1785937358.53 total 1051.68 last 19.85) - open/close 14064 (time 1785937374.29 total 1067.44 last 21.13) - open/close 14264 (time 1785937384.29 total 1077.44 last 20.00) - open/close 14416 (time 1785937394.56 total 1087.71 last 14.80) - open/close 14474 (time 1785937405.06 total 1098.21 last 5.53) - open/close 14581 (time 1785937415.38 total 1108.53 last 10.36) - open/close 14728 (time 1785937425.39 total 1118.54 last 14.69) - open/close 14865 (time 1785937435.40 total 1128.55 last 13.69) - open/close 14986 (time 1785937445.44 total 1138.59 last 12.05) - open/close 15087 (time 1785937455.65 total 1148.80 last 9.90) - open/close 15202 (time 1785937465.65 total 1158.80 last 11.49) - open/close 15296 (time 1785937476.70 total 1169.85 last 8.51) - open/close 15482 (time 1785937487.76 total 1180.91 last 16.81) - open/close 15600 (time 1785937498.76 total 1191.91 last 10.73) - open/close 15699 (time 1785937509.75 total 1202.90 last 9.01) - open/close 15864 (time 1785937520.28 total 1213.43 last 15.67) - open/close 15975 (time 1785937530.31 total 1223.46 last 11.07) - open/close 16241 (time 1785937541.24 total 1234.39 last 24.33) - open/close 16392 (time 1785937551.69 total 1244.84 last 14.45) - open/close 16498 (time 1785937562.31 total 1255.46 last 9.98) - open/close 16709 (time 1785937572.35 total 1265.50 last 21.01) - open/close 17190 (time 1785937582.35 total 1275.50 last 48.08) - open/close 17709 (time 1785937592.36 total 1285.51 last 51.85) - open/close 18461 (time 1785937602.37 total 1295.52 last 75.14) - open/close 18985 (time 1785937612.38 total 1305.53 last 52.33) - open/close 19498 (time 1785937622.40 total 1315.54 last 51.23) - open/close 20000 (time 1785937629.10 total 1322.25 last 74.90) - open/close 20569 (time 1785937639.10 total 1332.25 last 56.90) - open/close 21123 (time 1785937649.12 total 1342.27 last 55.27) - open/close 21813 (time 1785937659.13 total 1352.28 last 68.92) - open/close 22309 (time 1785937669.13 total 1362.28 last 49.60) - open/close 23006 (time 1785937679.14 total 1372.29 last 69.67) - open/close 23529 (time 1785937689.14 total 1382.29 last 52.28) - open/close 24340 (time 1785937699.14 total 1392.29 last 81.08) - open/close 24919 (time 1785937709.15 total 1402.30 last 57.87) - open/close 25707 (time 1785937719.15 total 1412.30 last 78.77) - open/close 26192 (time 1785937729.16 total 1422.31 last 48.46) - open/close 26672 (time 1785937739.16 total 1432.31 last 48.00) - open/close 27544 (time 1785937749.16 total 1442.31 last 87.20) - open/close 28391 (time 1785937759.18 total 1452.33 last 84.55) - open/close 28952 (time 1785937769.20 total 1462.35 last 55.96) - open/close 29413 (time 1785937779.21 total 1472.36 last 46.07) - open/close 30000 (time 1785937785.49 total 1478.64 last 93.41) - open/close 30694 (time 1785937795.51 total 1488.66 last 69.31) - open/close 31344 (time 1785937805.51 total 1498.66 last 64.94) - open/close 31821 (time 1785937815.51 total 1508.66 last 47.70) - open/close 32428 (time 1785937825.52 total 1518.67 last 60.69) - open/close 33158 (time 1785937835.52 total 1528.67 last 72.98) - open/close 33530 (time 1785937845.53 total 1538.68 last 37.17) - open/close 34130 (time 1785937855.54 total 1548.69 last 59.95) - open/close 34477 (time 1785937865.54 total 1558.69 last 34.67) - open/close 35093 (time 1785937875.55 total 1568.70 last 61.57) - open/close 35698 (time 1785937885.57 total 1578.72 last 60.37) - open/close 36469 (time 1785937895.57 total 1588.72 last 77.07) - open/close 37366 (time 1785937905.58 total 1598.73 last 89.66) - open/close 38367 (time 1785937915.59 total 1608.74 last 99.97) - open/close 39267 (time 1785937925.63 total 1618.78 last 89.63) - open/close 40000 (time 1785937935.18 total 1628.33 last 76.79) - open/close 40675 (time 1785937945.19 total 1638.34 last 67.42) - open/close 41129 (time 1785937955.21 total 1648.36 last 45.31) - open/close 41678 (time 1785937965.22 total 1658.37 last 54.85) - open/close 42208 (time 1785937975.22 total 1668.37 last 53.00) - open/close 42663 (time 1785937985.26 total 1678.41 last 45.33) - open/close 43236 (time 1785937995.30 total 1688.45 last 57.08) - open/close 43892 (time 1785938005.30 total 1698.45 last 65.60) - open/close 44589 (time 1785938015.31 total 1708.46 last 69.66) - open/close 45239 (time 1785938025.31 total 1718.46 last 64.98) - open/close 45810 (time 1785938035.32 total 1728.47 last 57.01) - open/close 46241 (time 1785938045.35 total 1738.50 last 43.01) - open/close 46919 (time 1785938055.36 total 1748.51 last 67.73) - open/close 47458 (time 1785938065.37 total 1758.52 last 53.84) - open/close 47885 (time 1785938075.42 total 1768.57 last 42.49) - open/close 48608 (time 1785938085.42 total 1778.57 last 72.27) - open/close 49237 (time 1785938095.43 total 1788.58 last 62.86) total: 50000 open/close in 1798.10 seconds: 27.81 ops/second - unlinked 0 (time 1785938110 ; total 0 ; last 0) - unlinked 10000 (time 1785938269 ; total 159 ; last 159) - unlinked 20000 (time 1785938434 ; total 324 ; last 165) - unlinked 30000 (time 1785938597 ; total 487 ; last 163) - unlinked 40000 (time 1785938754 ; total 644 ; last 157) total: 50000 unlinks in 792 seconds: 63.131313 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2