== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:35:07 (1781228107) start mds service on oleg120-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg120-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1781228190.60 total 10.35 last 13.43) - open/close 220 (time 1781228201.56 total 21.31 last 7.39) - open/close 280 (time 1781228212.63 total 32.38 last 5.42) - open/close 420 (time 1781228222.64 total 42.38 last 13.99) - open/close 494 (time 1781228232.90 total 52.65 last 7.21) - open/close 637 (time 1781228242.90 total 62.65 last 14.30) - open/close 766 (time 1781228255.43 total 75.18 last 10.30) - open/close 897 (time 1781228265.47 total 85.22 last 13.04) - open/close 1158 (time 1781228275.49 total 95.24 last 26.04) - open/close 1394 (time 1781228285.90 total 105.65 last 22.67) - open/close 1574 (time 1781228295.96 total 115.71 last 17.91) - open/close 1680 (time 1781228306.58 total 126.32 last 9.98) - open/close 1807 (time 1781228316.62 total 136.37 last 12.64) - open/close 1974 (time 1781228326.65 total 146.40 last 16.65) - open/close 2212 (time 1781228342.34 total 162.09 last 15.17) - open/close 2339 (time 1781228355.42 total 175.16 last 9.71) - open/close 2559 (time 1781228365.44 total 185.18 last 21.96) - open/close 2740 (time 1781228375.50 total 195.25 last 17.99) - open/close 2952 (time 1781228385.51 total 205.26 last 21.17) - open/close 3073 (time 1781228395.52 total 215.27 last 12.09) - open/close 3196 (time 1781228405.53 total 225.28 last 12.29) - open/close 3339 (time 1781228416.55 total 236.30 last 12.97) - open/close 3530 (time 1781228427.61 total 247.36 last 17.28) - open/close 3607 (time 1781228437.91 total 257.66 last 7.47) - open/close 3852 (time 1781228448.65 total 268.39 last 22.83) - open/close 3907 (time 1781228458.65 total 278.40 last 5.50) - open/close 4122 (time 1781228469.01 total 288.76 last 20.75) - open/close 4181 (time 1781228479.82 total 299.57 last 5.46) - open/close 4310 (time 1781228489.85 total 309.59 last 12.87) - open/close 4449 (time 1781228502.86 total 322.61 last 10.68) - open/close 4664 (time 1781228512.89 total 332.64 last 21.44) - open/close 4826 (time 1781228523.91 total 343.66 last 14.70) - open/close 4991 (time 1781228534.61 total 354.35 last 15.43) - open/close 5190 (time 1781228545.55 total 365.30 last 18.18) - open/close 5288 (time 1781228555.62 total 375.37 last 9.74) - open/close 5397 (time 1781228565.98 total 385.73 last 10.51) - open/close 5535 (time 1781228576.02 total 395.77 last 13.75) - open/close 5664 (time 1781228587.03 total 406.78 last 11.71) - open/close 5786 (time 1781228597.78 total 417.52 last 11.36) - open/close 6029 (time 1781228607.78 total 427.53 last 24.29) - open/close 6254 (time 1781228620.11 total 439.86 last 18.25) - open/close 6406 (time 1781228630.67 total 450.41 last 14.40) - open/close 6623 (time 1781228641.37 total 461.12 last 20.27) - open/close 6861 (time 1781228651.38 total 471.13 last 23.77) - open/close 7038 (time 1781228662.36 total 482.10 last 16.13) - open/close 7199 (time 1781228674.94 total 494.69 last 12.79) - open/close 7295 (time 1781228685.45 total 505.20 last 9.13) - open/close 7381 (time 1781228696.38 total 516.13 last 7.87) - open/close 7563 (time 1781228706.40 total 526.15 last 18.18) - open/close 7748 (time 1781228716.43 total 536.17 last 18.45) - open/close 7871 (time 1781228727.51 total 547.26 last 11.09) - open/close 8027 (time 1781228737.51 total 557.26 last 15.60) - open/close 8196 (time 1781228747.55 total 567.29 last 16.85) - open/close 8406 (time 1781228757.55 total 577.30 last 20.99) - open/close 8609 (time 1781228768.16 total 587.91 last 19.13) - open/close 8846 (time 1781228779.16 total 598.91 last 21.54) - open/close 8949 (time 1781228789.58 total 609.33 last 9.88) - open/close 9136 (time 1781228799.60 total 619.35 last 18.66) - open/close 9345 (time 1781228810.05 total 629.80 last 20.01) - open/close 9584 (time 1781228820.06 total 639.81 last 23.88) - open/close 9771 (time 1781228830.06 total 649.81 last 18.69) - open/close 10000 (time 1781228839.24 total 658.98 last 24.96) - open/close 10139 (time 1781228849.48 total 669.23 last 13.56) - open/close 10292 (time 1781228860.73 total 680.48 last 13.60) - open/close 10494 (time 1781228872.75 total 692.49 last 16.82) - open/close 10649 (time 1781228883.76 total 703.51 last 14.07) - open/close 10775 (time 1781228893.78 total 713.53 last 12.57) - open/close 10901 (time 1781228904.78 total 724.53 last 11.45) - open/close 11015 (time 1781228914.80 total 734.54 last 11.39) - open/close 11139 (time 1781228925.07 total 744.82 last 12.06) - open/close 11262 (time 1781228935.09 total 754.84 last 12.28) - open/close 11486 (time 1781228945.15 total 764.90 last 22.27) - open/close 11619 (time 1781228955.79 total 775.54 last 12.50) - open/close 11749 (time 1781228969.83 total 789.58 last 9.26) - open/close 11980 (time 1781228979.86 total 799.60 last 23.05) - open/close 12064 (time 1781228990.05 total 809.79 last 8.24) - open/close 12181 (time 1781229004.46 total 824.21 last 8.12) - open/close 12371 (time 1781229014.55 total 834.30 last 18.83) - open/close 12580 (time 1781229030.55 total 850.30 last 13.06) - open/close 12743 (time 1781229040.56 total 860.31 last 16.28) - open/close 12917 (time 1781229050.82 total 870.57 last 16.96) - open/close 13036 (time 1781229061.60 total 881.34 last 11.04) - open/close 13194 (time 1781229071.94 total 891.69 last 15.28) - open/close 13283 (time 1781229082.43 total 902.18 last 8.48) - open/close 13438 (time 1781229095.22 total 914.97 last 12.12) - open/close 13504 (time 1781229105.23 total 924.98 last 6.59) - open/close 13638 (time 1781229115.23 total 934.98 last 13.40) - open/close 13838 (time 1781229125.26 total 945.00 last 19.96) - open/close 14002 (time 1781229136.15 total 955.90 last 15.06) - open/close 14165 (time 1781229146.16 total 965.91 last 16.28) - open/close 14342 (time 1781229156.36 total 976.11 last 17.36) - open/close 14518 (time 1781229166.73 total 986.48 last 16.97) - open/close 14673 (time 1781229177.87 total 997.62 last 13.91) - open/close 14811 (time 1781229188.25 total 1008.00 last 13.30) - open/close 15027 (time 1781229198.99 total 1018.74 last 20.10) - open/close 15202 (time 1781229209.28 total 1029.03 last 17.00) - open/close 15311 (time 1781229219.85 total 1039.60 last 10.31) - open/close 15412 (time 1781229230.79 total 1050.54 last 9.23) - open/close 15497 (time 1781229242.05 total 1061.80 last 7.55) - open/close 15877 (time 1781229252.05 total 1071.80 last 37.98) - open/close 16539 (time 1781229262.07 total 1081.82 last 66.05) - open/close 17206 (time 1781229272.09 total 1091.84 last 66.61) - open/close 17981 (time 1781229282.10 total 1101.85 last 77.40) - open/close 18727 (time 1781229292.10 total 1111.85 last 74.59) - open/close 19416 (time 1781229302.12 total 1121.87 last 68.79) - open/close 20000 (time 1781229310.11 total 1129.85 last 73.10) - open/close 20719 (time 1781229320.11 total 1139.86 last 71.88) - open/close 21378 (time 1781229330.12 total 1149.87 last 65.83) - open/close 22027 (time 1781229340.12 total 1159.87 last 64.86) - open/close 22735 (time 1781229350.13 total 1169.88 last 70.76) - open/close 23459 (time 1781229360.15 total 1179.90 last 72.28) - open/close 24140 (time 1781229370.17 total 1189.92 last 67.95) - open/close 24799 (time 1781229380.20 total 1199.95 last 65.69) - open/close 25445 (time 1781229390.21 total 1209.95 last 64.57) - open/close 26172 (time 1781229400.22 total 1219.97 last 72.62) - open/close 26689 (time 1781229410.23 total 1229.98 last 51.63) - open/close 27141 (time 1781229420.25 total 1240.00 last 45.10) - open/close 27800 (time 1781229430.25 total 1250.00 last 65.89) - open/close 28596 (time 1781229440.26 total 1260.01 last 79.54) - open/close 29344 (time 1781229450.28 total 1270.03 last 74.65) - open/close 30000 (time 1781229459.69 total 1279.44 last 69.73) - open/close 30661 (time 1781229469.69 total 1289.44 last 66.08) - open/close 31398 (time 1781229479.70 total 1299.45 last 73.65) - open/close 32149 (time 1781229489.70 total 1309.45 last 75.08) - open/close 32793 (time 1781229499.71 total 1319.46 last 64.35) - open/close 33358 (time 1781229509.71 total 1329.46 last 56.49) - open/close 34109 (time 1781229519.72 total 1339.47 last 75.03) - open/close 34724 (time 1781229529.73 total 1349.47 last 61.47) - open/close 35329 (time 1781229539.74 total 1359.48 last 60.44) - open/close 35923 (time 1781229549.75 total 1369.49 last 59.34) - open/close 36545 (time 1781229559.75 total 1379.50 last 62.18) - open/close 37111 (time 1781229569.79 total 1389.54 last 56.37) - open/close 37711 (time 1781229579.79 total 1399.54 last 60.00) - open/close 38409 (time 1781229589.80 total 1409.55 last 69.75) - open/close 39147 (time 1781229599.81 total 1419.56 last 73.72) - open/close 39791 (time 1781229609.82 total 1429.57 last 64.34) - open/close 40000 (time 1781229613.13 total 1432.88 last 63.08) - open/close 40628 (time 1781229623.14 total 1442.88 last 62.78) - open/close 41273 (time 1781229633.15 total 1452.90 last 64.41) - open/close 41899 (time 1781229643.15 total 1462.90 last 62.58) - open/close 42473 (time 1781229653.18 total 1472.93 last 57.25) - open/close 43030 (time 1781229663.19 total 1482.94 last 55.66) - open/close 43548 (time 1781229673.21 total 1492.96 last 51.67) - open/close 44226 (time 1781229683.22 total 1502.97 last 67.76) - open/close 44848 (time 1781229693.22 total 1512.97 last 62.17) - open/close 45518 (time 1781229703.23 total 1522.98 last 66.94) - open/close 46225 (time 1781229713.30 total 1533.05 last 70.21) - open/close 46828 (time 1781229723.39 total 1543.13 last 59.81) - open/close 47574 (time 1781229733.39 total 1553.14 last 74.58) - open/close 48160 (time 1781229743.39 total 1563.14 last 58.60) - open/close 48816 (time 1781229753.40 total 1573.15 last 65.52) - open/close 49392 (time 1781229763.45 total 1583.20 last 57.33) - open/close 49954 (time 1781229773.47 total 1593.22 last 56.09) total: 50000 open/close in 1594.01 seconds: 31.37 ops/second - unlinked 0 (time 1781229779 ; total 0 ; last 0) - unlinked 10000 (time 1781229950 ; total 171 ; last 171) - unlinked 20000 (time 1781230114 ; total 335 ; last 164) - unlinked 30000 (time 1781230274 ; total 495 ; last 160) - unlinked 40000 (time 1781230443 ; total 664 ; last 169) total: 50000 unlinks in 821 seconds: 60.901340 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2