== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:31:37 (1774877497) start mds service on oleg455-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg455-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 247 (time 1774877576.20 total 10.01 last 24.69) - open/close 429 (time 1774877587.00 total 20.81 last 16.85) - open/close 612 (time 1774877597.01 total 30.82 last 18.27) - open/close 819 (time 1774877607.02 total 40.83 last 20.69) - open/close 1022 (time 1774877617.06 total 50.87 last 20.22) - open/close 1275 (time 1774877627.62 total 61.43 last 23.94) - open/close 1420 (time 1774877637.63 total 71.44 last 14.49) - open/close 1636 (time 1774877647.63 total 81.44 last 21.59) - open/close 1897 (time 1774877658.55 total 92.36 last 23.90) - open/close 2076 (time 1774877671.25 total 105.06 last 14.10) - open/close 2335 (time 1774877681.25 total 115.06 last 25.89) - open/close 2523 (time 1774877691.72 total 125.53 last 17.96) - open/close 2644 (time 1774877701.72 total 135.53 last 12.09) - open/close 2802 (time 1774877712.63 total 146.44 last 14.49) - open/close 2937 (time 1774877722.64 total 156.45 last 13.49) - open/close 3066 (time 1774877732.65 total 166.46 last 12.89) - open/close 3248 (time 1774877743.15 total 176.96 last 17.33) - open/close 3328 (time 1774877754.05 total 187.86 last 7.34) - open/close 3528 (time 1774877765.11 total 198.92 last 18.07) - open/close 3771 (time 1774877775.12 total 208.93 last 24.28) - open/close 3980 (time 1774877785.40 total 219.21 last 20.33) - open/close 4233 (time 1774877795.41 total 229.22 last 25.27) - open/close 4494 (time 1774877805.44 total 239.24 last 26.04) - open/close 4680 (time 1774877815.45 total 249.26 last 18.58) - open/close 4892 (time 1774877826.47 total 260.28 last 19.23) - open/close 5055 (time 1774877837.76 total 271.57 last 14.44) - open/close 5205 (time 1774877847.94 total 281.75 last 14.73) - open/close 5460 (time 1774877857.95 total 291.76 last 25.47) - open/close 5751 (time 1774877867.96 total 301.77 last 29.07) - open/close 5981 (time 1774877878.47 total 312.28 last 21.89) - open/close 6200 (time 1774877888.48 total 322.29 last 21.87) - open/close 6419 (time 1774877898.49 total 332.30 last 21.89) - open/close 6643 (time 1774877909.19 total 343.00 last 20.92) - open/close 6867 (time 1774877919.53 total 353.34 last 21.68) - open/close 7123 (time 1774877931.72 total 365.53 last 21.00) - open/close 7321 (time 1774877941.73 total 375.54 last 19.78) - open/close 7564 (time 1774877952.36 total 386.17 last 22.85) - open/close 7762 (time 1774877962.61 total 396.42 last 19.32) - open/close 7962 (time 1774877972.62 total 406.43 last 19.98) - open/close 8104 (time 1774877982.62 total 416.43 last 14.20) - open/close 8281 (time 1774877993.01 total 426.82 last 17.03) - open/close 8335 (time 1774878007.41 total 441.22 last 3.75) - open/close 8603 (time 1774878017.64 total 451.45 last 26.20) - open/close 8878 (time 1774878027.65 total 461.45 last 27.49) - open/close 9022 (time 1774878037.99 total 471.80 last 13.92) - open/close 9225 (time 1774878048.01 total 481.82 last 20.27) - open/close 9399 (time 1774878060.46 total 494.27 last 13.97) - open/close 9625 (time 1774878070.47 total 504.28 last 22.58) - open/close 9818 (time 1774878080.48 total 514.29 last 19.28) - open/close 10000 (time 1774878090.49 total 524.30 last 18.18) - open/close 10228 (time 1774878100.52 total 534.33 last 22.73) - open/close 10383 (time 1774878110.66 total 544.47 last 15.29) - open/close 10564 (time 1774878121.51 total 555.32 last 16.68) - open/close 10688 (time 1774878131.53 total 565.34 last 12.38) - open/close 10905 (time 1774878147.44 total 581.25 last 13.64) - open/close 10986 (time 1774878158.51 total 592.32 last 7.31) - open/close 11250 (time 1774878168.52 total 602.33 last 26.38) - open/close 11339 (time 1774878180.71 total 614.52 last 7.30) - open/close 11476 (time 1774878190.90 total 624.71 last 13.44) - open/close 11693 (time 1774878200.92 total 634.73 last 21.65) - open/close 11825 (time 1774878211.06 total 644.87 last 13.02) - open/close 11913 (time 1774878221.68 total 655.49 last 8.29) - open/close 11982 (time 1774878231.69 total 665.50 last 6.89) - open/close 12087 (time 1774878241.72 total 675.53 last 10.47) - open/close 12227 (time 1774878251.75 total 685.56 last 13.96) - open/close 12416 (time 1774878261.76 total 695.56 last 18.88) - open/close 12525 (time 1774878272.01 total 705.82 last 10.63) - open/close 12600 (time 1774878283.46 total 717.27 last 6.55) - open/close 12732 (time 1774878293.47 total 727.28 last 13.20) - open/close 12868 (time 1774878303.73 total 737.54 last 13.25) - open/close 12899 (time 1774878314.02 total 747.83 last 3.01) - open/close 12926 (time 1774878328.06 total 761.87 last 1.92) - open/close 13037 (time 1774878339.00 total 772.81 last 10.15) - open/close 13229 (time 1774878349.34 total 783.15 last 18.57) - open/close 13504 (time 1774878360.18 total 793.99 last 25.36) - open/close 13759 (time 1774878370.63 total 804.44 last 24.41) - open/close 14025 (time 1774878380.66 total 814.47 last 26.51) - open/close 14093 (time 1774878390.66 total 824.47 last 6.80) - open/close 14306 (time 1774878401.12 total 834.93 last 20.36) - open/close 14520 (time 1774878411.99 total 845.80 last 19.70) - open/close 14652 (time 1774878422.38 total 856.19 last 12.70) - open/close 14677 (time 1774878434.71 total 868.52 last 2.03) - open/close 14708 (time 1774878444.85 total 878.66 last 3.06) - open/close 14779 (time 1774878454.85 total 888.66 last 7.10) - open/close 14908 (time 1774878464.86 total 898.67 last 12.89) - open/close 15055 (time 1774878474.89 total 908.70 last 14.66) - open/close 15304 (time 1774878484.90 total 918.71 last 24.88) - open/close 15435 (time 1774878494.93 total 928.74 last 13.06) - open/close 15654 (time 1774878504.93 total 938.74 last 21.90) - open/close 15743 (time 1774878515.43 total 949.24 last 8.48) - open/close 15939 (time 1774878526.32 total 960.13 last 18.00) - open/close 16143 (time 1774878536.83 total 970.64 last 19.40) - open/close 16389 (time 1774878546.85 total 980.66 last 24.56) - open/close 16527 (time 1774878556.86 total 990.67 last 13.79) - open/close 16759 (time 1774878566.87 total 1000.68 last 23.18) - open/close 16937 (time 1774878577.86 total 1011.67 last 16.19) - open/close 17373 (time 1774878587.88 total 1021.69 last 43.52) - open/close 18088 (time 1774878597.89 total 1031.70 last 71.44) - open/close 18804 (time 1774878607.89 total 1041.70 last 71.56) - open/close 19455 (time 1774878617.89 total 1051.70 last 65.09) - open/close 20000 (time 1774878626.57 total 1060.38 last 62.83) - open/close 20723 (time 1774878636.57 total 1070.38 last 72.29) - open/close 21386 (time 1774878646.58 total 1080.38 last 66.25) - open/close 22044 (time 1774878656.58 total 1090.39 last 65.78) - open/close 22709 (time 1774878666.58 total 1100.39 last 66.48) - open/close 23372 (time 1774878676.59 total 1110.40 last 66.24) - open/close 24055 (time 1774878686.60 total 1120.41 last 68.25) - open/close 24711 (time 1774878696.60 total 1130.41 last 65.56) - open/close 25423 (time 1774878706.61 total 1140.42 last 71.15) - open/close 26151 (time 1774878716.62 total 1150.43 last 72.75) - open/close 26838 (time 1774878726.63 total 1160.43 last 68.65) - open/close 27540 (time 1774878736.63 total 1170.44 last 70.19) - open/close 28180 (time 1774878746.63 total 1180.44 last 63.98) - open/close 28909 (time 1774878756.64 total 1190.45 last 72.79) - open/close 29588 (time 1774878766.65 total 1200.46 last 67.87) - open/close 30000 (time 1774878772.24 total 1206.05 last 73.74) - open/close 30723 (time 1774878782.24 total 1216.05 last 72.27) - open/close 31453 (time 1774878792.24 total 1226.05 last 72.98) - open/close 32055 (time 1774878802.27 total 1236.08 last 60.05) - open/close 32570 (time 1774878812.27 total 1246.08 last 51.49) - open/close 33170 (time 1774878822.27 total 1256.08 last 60.00) - open/close 33725 (time 1774878832.28 total 1266.09 last 55.45) - open/close 34368 (time 1774878842.29 total 1276.10 last 64.24) - open/close 35042 (time 1774878852.30 total 1286.11 last 67.31) - open/close 35718 (time 1774878862.33 total 1296.14 last 67.44) - open/close 36360 (time 1774878872.33 total 1306.14 last 64.16) - open/close 37059 (time 1774878882.35 total 1316.15 last 69.83) - open/close 37756 (time 1774878892.35 total 1326.16 last 69.67) - open/close 38372 (time 1774878902.36 total 1336.17 last 61.55) - open/close 38978 (time 1774878912.36 total 1346.17 last 60.58) - open/close 39691 (time 1774878922.37 total 1356.18 last 71.26) - open/close 40000 (time 1774878926.90 total 1360.71 last 68.12) - open/close 40679 (time 1774878936.91 total 1370.72 last 67.85) - open/close 41382 (time 1774878946.92 total 1380.73 last 70.23) - open/close 42056 (time 1774878956.94 total 1390.75 last 67.29) - open/close 42769 (time 1774878966.94 total 1400.75 last 71.26) - open/close 43447 (time 1774878976.95 total 1410.76 last 67.78) - open/close 44073 (time 1774878986.96 total 1420.77 last 62.54) - open/close 44839 (time 1774878996.96 total 1430.77 last 76.59) - open/close 45514 (time 1774879006.97 total 1440.78 last 67.41) - open/close 46213 (time 1774879016.98 total 1450.79 last 69.87) - open/close 46955 (time 1774879026.98 total 1460.79 last 74.15) - open/close 47667 (time 1774879036.99 total 1470.80 last 71.18) - open/close 48384 (time 1774879046.99 total 1480.80 last 71.68) - open/close 49104 (time 1774879057.00 total 1490.81 last 71.90) - open/close 49763 (time 1774879067.00 total 1500.81 last 65.89) total: 50000 open/close in 1504.58 seconds: 33.23 ops/second - unlinked 0 (time 1774879075 ; total 0 ; last 0) - unlinked 10000 (time 1774879230 ; total 155 ; last 155) - unlinked 20000 (time 1774879386 ; total 311 ; last 156) - unlinked 30000 (time 1774879545 ; total 470 ; last 159) - unlinked 40000 (time 1774879703 ; total 628 ; last 158) total: 50000 unlinks in 781 seconds: 64.020485 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2