== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:38:18 (1779129498) start mds service on oleg120-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg120-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1779129591.95 total 10.17 last 13.18) - open/close 238 (time 1779129601.96 total 20.18 last 10.38) - open/close 443 (time 1779129612.88 total 31.10 last 18.78) - open/close 641 (time 1779129622.88 total 41.10 last 19.80) - open/close 813 (time 1779129632.96 total 51.18 last 17.06) - open/close 971 (time 1779129643.52 total 61.75 last 14.96) - open/close 1158 (time 1779129653.54 total 71.76 last 18.67) - open/close 1351 (time 1779129663.59 total 81.81 last 19.20) - open/close 1478 (time 1779129673.62 total 91.84 last 12.66) - open/close 1602 (time 1779129684.04 total 102.26 last 11.90) - open/close 1691 (time 1779129694.53 total 112.75 last 8.49) - open/close 1793 (time 1779129704.56 total 122.78 last 10.17) - open/close 1892 (time 1779129715.53 total 133.75 last 9.02) - open/close 2041 (time 1779129725.77 total 143.99 last 14.55) - open/close 2108 (time 1779129736.72 total 154.94 last 6.12) - open/close 2226 (time 1779129746.88 total 165.10 last 11.61) - open/close 2253 (time 1779129757.95 total 176.17 last 2.44) - open/close 2331 (time 1779129768.06 total 186.28 last 7.71) - open/close 2509 (time 1779129778.08 total 196.30 last 17.77) - open/close 2612 (time 1779129789.08 total 207.30 last 9.37) - open/close 2768 (time 1779129799.94 total 218.16 last 14.37) - open/close 2895 (time 1779129809.95 total 228.17 last 12.69) - open/close 3103 (time 1779129819.95 total 238.17 last 20.80) - open/close 3315 (time 1779129829.95 total 248.17 last 21.19) - open/close 3485 (time 1779129839.96 total 258.18 last 16.99) - open/close 3597 (time 1779129850.11 total 268.33 last 11.04) - open/close 3842 (time 1779129860.31 total 278.53 last 24.02) - open/close 4009 (time 1779129871.37 total 289.59 last 15.10) - open/close 4282 (time 1779129881.37 total 299.59 last 27.29) - open/close 4414 (time 1779129891.38 total 309.60 last 13.19) - open/close 4641 (time 1779129901.40 total 319.63 last 22.65) - open/close 4919 (time 1779129912.44 total 330.66 last 25.20) - open/close 5174 (time 1779129922.51 total 340.74 last 25.30) - open/close 5325 (time 1779129933.36 total 351.58 last 13.92) - open/close 5477 (time 1779129944.07 total 362.29 last 14.19) - open/close 5604 (time 1779129954.09 total 372.31 last 12.68) - open/close 5794 (time 1779129964.68 total 382.90 last 17.95) - open/close 5962 (time 1779129974.71 total 392.93 last 16.75) - open/close 6122 (time 1779129984.72 total 402.94 last 15.98) - open/close 6347 (time 1779129994.99 total 413.22 last 21.91) - open/close 6608 (time 1779130005.00 total 423.22 last 26.09) - open/close 6789 (time 1779130015.17 total 433.39 last 17.80) - open/close 6984 (time 1779130025.18 total 443.40 last 19.47) - open/close 7158 (time 1779130035.22 total 453.44 last 17.33) - open/close 7192 (time 1779130045.44 total 463.66 last 3.33) - open/close 7221 (time 1779130055.44 total 473.66 last 2.90) - open/close 7395 (time 1779130066.29 total 484.51 last 16.03) - open/close 7526 (time 1779130076.31 total 494.53 last 13.08) - open/close 7671 (time 1779130086.68 total 504.90 last 13.98) - open/close 7824 (time 1779130098.17 total 516.39 last 13.31) - open/close 7967 (time 1779130108.76 total 526.98 last 13.50) - open/close 8203 (time 1779130118.79 total 537.01 last 23.53) - open/close 8364 (time 1779130129.09 total 547.31 last 15.63) - open/close 8506 (time 1779130139.10 total 557.32 last 14.18) - open/close 8738 (time 1779130149.10 total 567.32 last 23.20) - open/close 8848 (time 1779130159.82 total 578.05 last 10.26) - open/close 8964 (time 1779130169.94 total 588.16 last 11.47) - open/close 9090 (time 1779130180.22 total 598.44 last 12.25) - open/close 9221 (time 1779130191.10 total 609.32 last 12.04) - open/close 9390 (time 1779130203.66 total 621.88 last 13.46) - open/close 9562 (time 1779130213.67 total 631.89 last 17.19) - open/close 9799 (time 1779130224.21 total 642.43 last 22.47) - open/close 9941 (time 1779130234.89 total 653.11 last 13.30) - open/close 10000 (time 1779130240.37 total 658.59 last 10.78) - open/close 10044 (time 1779130250.88 total 669.10 last 4.19) - open/close 10164 (time 1779130260.89 total 679.11 last 11.98) - open/close 10293 (time 1779130274.23 total 692.45 last 9.67) - open/close 10394 (time 1779130284.61 total 702.83 last 9.73) - open/close 10603 (time 1779130295.31 total 713.53 last 19.53) - open/close 10820 (time 1779130305.32 total 723.54 last 21.69) - open/close 10857 (time 1779130315.97 total 734.19 last 3.47) - open/close 11001 (time 1779130326.00 total 744.22 last 14.36) - open/close 11136 (time 1779130336.00 total 754.22 last 13.49) - open/close 11281 (time 1779130346.91 total 765.13 last 13.29) - open/close 11433 (time 1779130356.92 total 775.14 last 15.19) - open/close 11632 (time 1779130367.27 total 785.50 last 19.22) - open/close 11810 (time 1779130377.31 total 795.53 last 17.74) - open/close 11868 (time 1779130387.40 total 805.62 last 5.75) - open/close 12040 (time 1779130397.85 total 816.07 last 16.46) - open/close 12210 (time 1779130408.23 total 826.45 last 16.38) - open/close 12462 (time 1779130418.25 total 836.47 last 25.15) - open/close 12610 (time 1779130428.37 total 846.59 last 14.62) - open/close 12811 (time 1779130440.69 total 858.91 last 16.32) - open/close 13000 (time 1779130452.95 total 871.17 last 15.41) - open/close 13085 (time 1779130463.63 total 881.86 last 7.96) - open/close 13144 (time 1779130474.70 total 892.92 last 5.33) - open/close 13297 (time 1779130484.74 total 902.96 last 15.23) - open/close 13464 (time 1779130494.75 total 912.98 last 16.67) - open/close 13619 (time 1779130506.31 total 924.53 last 13.41) - open/close 13846 (time 1779130517.26 total 935.48 last 20.73) - open/close 14025 (time 1779130527.27 total 945.49 last 17.89) - open/close 14224 (time 1779130537.92 total 956.14 last 18.67) - open/close 14399 (time 1779130548.32 total 966.55 last 16.82) - open/close 14498 (time 1779130558.99 total 977.21 last 9.28) - open/close 14612 (time 1779130569.00 total 987.22 last 11.39) - open/close 14754 (time 1779130579.71 total 997.93 last 13.25) - open/close 14879 (time 1779130592.51 total 1010.73 last 9.77) - open/close 14902 (time 1779130602.54 total 1020.76 last 2.29) - open/close 14974 (time 1779130612.61 total 1030.83 last 7.14) - open/close 15060 (time 1779130623.23 total 1041.45 last 8.10) - open/close 15278 (time 1779130633.25 total 1051.47 last 21.77) - open/close 15495 (time 1779130643.52 total 1061.74 last 21.12) - open/close 15667 (time 1779130653.53 total 1071.75 last 17.19) - open/close 15865 (time 1779130663.55 total 1081.77 last 19.76) - open/close 16497 (time 1779130673.56 total 1091.78 last 63.16) - open/close 17238 (time 1779130683.56 total 1101.78 last 74.08) - open/close 17743 (time 1779130693.56 total 1111.78 last 50.49) - open/close 18340 (time 1779130703.56 total 1121.78 last 59.69) - open/close 18898 (time 1779130713.57 total 1131.79 last 55.76) - open/close 19696 (time 1779130723.57 total 1141.80 last 79.74) - open/close 20000 (time 1779130727.70 total 1145.93 last 73.61) - open/close 20715 (time 1779130737.71 total 1155.93 last 71.48) - open/close 21417 (time 1779130747.71 total 1165.93 last 70.18) - open/close 22127 (time 1779130757.72 total 1175.94 last 70.94) - open/close 22904 (time 1779130767.73 total 1185.95 last 77.62) - open/close 23636 (time 1779130777.74 total 1195.96 last 73.15) - open/close 24337 (time 1779130787.74 total 1205.97 last 70.04) - open/close 24951 (time 1779130797.76 total 1215.98 last 61.32) - open/close 25605 (time 1779130807.76 total 1225.98 last 65.40) - open/close 26343 (time 1779130817.76 total 1235.98 last 73.79) - open/close 27091 (time 1779130827.76 total 1245.98 last 74.78) - open/close 27895 (time 1779130837.76 total 1255.99 last 80.38) - open/close 28664 (time 1779130847.79 total 1266.02 last 76.67) - open/close 29222 (time 1779130857.84 total 1276.06 last 55.55) - open/close 29789 (time 1779130867.84 total 1286.07 last 56.67) - open/close 30000 (time 1779130872.72 total 1290.94 last 43.31) - open/close 30437 (time 1779130882.74 total 1300.96 last 43.59) - open/close 31036 (time 1779130892.75 total 1310.97 last 59.85) - open/close 31666 (time 1779130902.75 total 1320.97 last 62.99) - open/close 32379 (time 1779130912.77 total 1330.99 last 71.20) - open/close 32925 (time 1779130922.77 total 1340.99 last 54.55) - open/close 33515 (time 1779130932.80 total 1351.02 last 58.84) - open/close 34161 (time 1779130942.83 total 1361.05 last 64.40) - open/close 34766 (time 1779130952.84 total 1371.06 last 60.47) - open/close 35466 (time 1779130962.85 total 1381.07 last 69.93) - open/close 36121 (time 1779130972.85 total 1391.07 last 65.49) - open/close 36784 (time 1779130982.87 total 1401.09 last 66.13) - open/close 37472 (time 1779130992.88 total 1411.10 last 68.76) - open/close 38222 (time 1779131002.89 total 1421.11 last 74.92) - open/close 38881 (time 1779131012.90 total 1431.12 last 65.80) - open/close 39587 (time 1779131022.91 total 1441.13 last 70.53) - open/close 40000 (time 1779131028.04 total 1446.26 last 80.57) - open/close 40842 (time 1779131038.04 total 1456.26 last 84.19) - open/close 41682 (time 1779131048.04 total 1466.26 last 84.00) - open/close 42502 (time 1779131058.07 total 1476.29 last 81.75) - open/close 43221 (time 1779131068.08 total 1486.30 last 71.81) - open/close 43955 (time 1779131078.08 total 1496.31 last 73.40) - open/close 44597 (time 1779131088.09 total 1506.31 last 64.16) - open/close 45258 (time 1779131098.09 total 1516.32 last 66.08) - open/close 45995 (time 1779131108.10 total 1526.32 last 73.64) - open/close 46669 (time 1779131118.11 total 1536.33 last 67.36) - open/close 47311 (time 1779131128.11 total 1546.33 last 64.17) - open/close 48081 (time 1779131138.13 total 1556.35 last 76.86) - open/close 48794 (time 1779131148.13 total 1566.35 last 71.29) - open/close 49344 (time 1779131158.14 total 1576.36 last 54.94) total: 50000 open/close in 1585.96 seconds: 31.53 ops/second - unlinked 0 (time 1779131173 ; total 0 ; last 0) - unlinked 10000 (time 1779131320 ; total 147 ; last 147) - unlinked 20000 (time 1779131464 ; total 291 ; last 144) - unlinked 30000 (time 1779131614 ; total 441 ; last 150) - unlinked 40000 (time 1779131772 ; total 599 ; last 158) total: 50000 unlinks in 769 seconds: 65.019508 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: modules unloaded. pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2