== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:46:23 (1786801583) start mds service on oleg302-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg302-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1786801666.72 total 10.00 last 23.40) - open/close 469 (time 1786801676.73 total 20.01 last 23.47) - open/close 635 (time 1786801686.79 total 30.07 last 16.50) - open/close 843 (time 1786801698.24 total 41.52 last 18.17) - open/close 1068 (time 1786801708.85 total 52.14 last 21.19) - open/close 1289 (time 1786801718.86 total 62.15 last 22.08) - open/close 1537 (time 1786801728.87 total 72.16 last 24.78) - open/close 1746 (time 1786801738.88 total 82.17 last 20.88) - open/close 1910 (time 1786801748.89 total 92.18 last 16.39) - open/close 2135 (time 1786801759.87 total 103.15 last 20.50) - open/close 2384 (time 1786801770.43 total 113.71 last 23.57) - open/close 2636 (time 1786801780.82 total 124.11 last 24.24) - open/close 2786 (time 1786801791.58 total 134.87 last 13.94) - open/close 2966 (time 1786801801.60 total 144.89 last 17.97) - open/close 3092 (time 1786801811.63 total 154.91 last 12.57) - open/close 3358 (time 1786801821.63 total 164.91 last 26.60) - open/close 3590 (time 1786801831.64 total 174.93 last 23.16) - open/close 3712 (time 1786801841.66 total 184.94 last 12.18) - open/close 3832 (time 1786801851.74 total 195.02 last 11.91) - open/close 4023 (time 1786801862.46 total 205.74 last 17.81) - open/close 4244 (time 1786801873.40 total 216.69 last 20.19) - open/close 4484 (time 1786801883.42 total 226.70 last 23.96) - open/close 4752 (time 1786801893.91 total 237.19 last 25.56) - open/close 4916 (time 1786801903.92 total 247.20 last 16.38) - open/close 5159 (time 1786801914.89 total 258.17 last 22.16) - open/close 5346 (time 1786801925.41 total 268.69 last 17.77) - open/close 5453 (time 1786801935.42 total 278.70 last 10.69) - open/close 5714 (time 1786801945.44 total 288.72 last 26.06) - open/close 5944 (time 1786801955.45 total 298.73 last 22.97) - open/close 6172 (time 1786801965.46 total 308.75 last 22.77) - open/close 6407 (time 1786801975.90 total 319.18 last 22.51) - open/close 6623 (time 1786801986.31 total 329.59 last 20.76) - open/close 6878 (time 1786801997.08 total 340.36 last 23.67) - open/close 7099 (time 1786802007.36 total 350.64 last 21.50) - open/close 7312 (time 1786802017.73 total 361.01 last 20.55) - open/close 7581 (time 1786802028.74 total 372.03 last 24.42) - open/close 7849 (time 1786802038.76 total 382.05 last 26.74) - open/close 8106 (time 1786802048.78 total 392.07 last 25.65) - open/close 8322 (time 1786802058.98 total 402.26 last 21.19) - open/close 8580 (time 1786802068.98 total 412.27 last 25.79) - open/close 8826 (time 1786802079.87 total 423.15 last 22.60) - open/close 9053 (time 1786802090.49 total 433.77 last 21.37) - open/close 9275 (time 1786802101.25 total 444.54 last 20.63) - open/close 9492 (time 1786802111.26 total 454.54 last 21.69) - open/close 9729 (time 1786802121.70 total 464.98 last 22.70) - open/close 9930 (time 1786802132.50 total 475.78 last 18.61) - open/close 10000 (time 1786802134.86 total 478.14 last 29.67) - open/close 10233 (time 1786802145.28 total 488.57 last 22.35) - open/close 10451 (time 1786802155.84 total 499.13 last 20.64) - open/close 10642 (time 1786802166.80 total 510.08 last 17.43) - open/close 10887 (time 1786802177.21 total 520.50 last 23.52) - open/close 11170 (time 1786802187.21 total 530.50 last 28.30) - open/close 11389 (time 1786802197.23 total 540.51 last 21.87) - open/close 11566 (time 1786802208.20 total 551.48 last 16.13) - open/close 11704 (time 1786802218.22 total 561.50 last 13.77) - open/close 11853 (time 1786802229.13 total 572.41 last 13.66) - open/close 11999 (time 1786802239.15 total 582.44 last 14.56) - open/close 12225 (time 1786802250.64 total 593.93 last 19.67) - open/close 12499 (time 1786802260.65 total 603.93 last 27.39) - open/close 12712 (time 1786802270.81 total 614.10 last 20.95) - open/close 12930 (time 1786802280.82 total 624.10 last 21.79) - open/close 12975 (time 1786802290.88 total 634.17 last 4.47) - open/close 13179 (time 1786802301.89 total 645.18 last 18.53) - open/close 13378 (time 1786802311.90 total 655.18 last 19.90) - open/close 13623 (time 1786802321.94 total 665.23 last 24.39) - open/close 13760 (time 1786802331.95 total 675.24 last 13.69) - open/close 13986 (time 1786802342.87 total 686.15 last 20.71) - open/close 14182 (time 1786802353.49 total 696.77 last 18.46) - open/close 14306 (time 1786802363.52 total 706.81 last 12.35) - open/close 14476 (time 1786802374.34 total 717.62 last 15.72) - open/close 14650 (time 1786802385.10 total 728.38 last 16.17) - open/close 14859 (time 1786802395.27 total 738.55 last 20.55) - open/close 15058 (time 1786802405.76 total 749.04 last 18.97) - open/close 15356 (time 1786802415.77 total 759.06 last 29.75) - open/close 15554 (time 1786802425.79 total 769.07 last 19.78) - open/close 15805 (time 1786802435.80 total 779.08 last 25.08) - open/close 15950 (time 1786802445.80 total 789.09 last 14.49) - open/close 16087 (time 1786802456.47 total 799.76 last 12.84) - open/close 16288 (time 1786802466.50 total 809.78 last 20.05) - open/close 16394 (time 1786802476.64 total 819.92 last 10.45) - open/close 16591 (time 1786802486.64 total 829.93 last 19.69) - open/close 16771 (time 1786802496.66 total 839.95 last 17.97) - open/close 16919 (time 1786802507.50 total 850.78 last 13.66) - open/close 17084 (time 1786802517.96 total 861.24 last 15.78) - open/close 17268 (time 1786802528.34 total 871.62 last 17.73) - open/close 17482 (time 1786802538.36 total 881.64 last 21.35) - open/close 17700 (time 1786802549.32 total 892.61 last 19.89) - open/close 17943 (time 1786802559.33 total 902.61 last 24.29) - open/close 18124 (time 1786802569.33 total 912.62 last 18.09) - open/close 18372 (time 1786802579.34 total 922.62 last 24.79) - open/close 18576 (time 1786802590.62 total 933.90 last 18.08) - open/close 18822 (time 1786802600.93 total 944.21 last 23.86) - open/close 19098 (time 1786802611.71 total 955.00 last 25.59) - open/close 19353 (time 1786802621.72 total 965.00 last 25.48) - open/close 19705 (time 1786802631.72 total 975.00 last 35.20) - open/close 20000 (time 1786802635.74 total 979.02 last 73.45) - open/close 20701 (time 1786802645.74 total 989.02 last 70.08) - open/close 21295 (time 1786802655.74 total 999.02 last 59.39) - open/close 22040 (time 1786802665.75 total 1009.03 last 74.46) - open/close 22746 (time 1786802675.77 total 1019.05 last 70.47) - open/close 23500 (time 1786802685.77 total 1029.05 last 75.37) - open/close 24268 (time 1786802695.77 total 1039.05 last 76.79) - open/close 25041 (time 1786802705.77 total 1049.05 last 77.29) - open/close 25856 (time 1786802715.78 total 1059.07 last 81.40) - open/close 26557 (time 1786802725.79 total 1069.08 last 70.03) - open/close 27350 (time 1786802735.80 total 1079.08 last 79.26) - open/close 28089 (time 1786802745.81 total 1089.09 last 73.82) - open/close 28868 (time 1786802755.82 total 1099.11 last 77.78) - open/close 29619 (time 1786802765.83 total 1109.12 last 75.04) - open/close 30000 (time 1786802770.51 total 1113.79 last 81.52) - open/close 30750 (time 1786802780.51 total 1123.79 last 74.99) - open/close 31533 (time 1786802790.52 total 1133.80 last 78.20) - open/close 32224 (time 1786802800.53 total 1143.81 last 69.06) - open/close 32931 (time 1786802810.54 total 1153.82 last 70.60) - open/close 33705 (time 1786802820.54 total 1163.83 last 77.36) - open/close 34382 (time 1786802830.56 total 1173.85 last 67.56) - open/close 35032 (time 1786802840.58 total 1183.86 last 64.93) - open/close 35704 (time 1786802850.59 total 1193.87 last 67.11) - open/close 36394 (time 1786802860.59 total 1203.87 last 68.99) - open/close 36904 (time 1786802870.59 total 1213.88 last 50.98) - open/close 37514 (time 1786802880.60 total 1223.88 last 60.97) - open/close 38147 (time 1786802890.60 total 1233.89 last 63.29) - open/close 38795 (time 1786802900.61 total 1243.89 last 64.76) - open/close 39505 (time 1786802910.61 total 1253.89 last 70.98) - open/close 40000 (time 1786802917.76 total 1261.04 last 69.23) - open/close 40600 (time 1786802927.79 total 1271.07 last 59.83) - open/close 41169 (time 1786802937.79 total 1281.08 last 56.87) - open/close 41857 (time 1786802947.79 total 1291.08 last 68.80) - open/close 42533 (time 1786802957.80 total 1301.08 last 67.59) - open/close 43195 (time 1786802967.80 total 1311.08 last 66.17) - open/close 43815 (time 1786802977.81 total 1321.09 last 61.94) - open/close 44447 (time 1786802987.81 total 1331.09 last 63.19) - open/close 45149 (time 1786802997.82 total 1341.10 last 70.14) - open/close 45729 (time 1786803007.82 total 1351.11 last 57.99) - open/close 46374 (time 1786803017.83 total 1361.12 last 64.42) - open/close 47059 (time 1786803027.85 total 1371.14 last 68.37) - open/close 47733 (time 1786803037.86 total 1381.14 last 67.37) - open/close 48418 (time 1786803047.86 total 1391.14 last 68.49) - open/close 49062 (time 1786803057.87 total 1401.15 last 64.34) - open/close 49710 (time 1786803067.87 total 1411.15 last 64.78) total: 50000 open/close in 1415.26 seconds: 35.33 ops/second - unlinked 0 (time 1786803076 ; total 0 ; last 0) - unlinked 10000 (time 1786803221 ; total 145 ; last 145) - unlinked 20000 (time 1786803371 ; total 295 ; last 150) - unlinked 30000 (time 1786803505 ; total 429 ; last 134) - unlinked 40000 (time 1786803650 ; total 574 ; last 145) total: 50000 unlinks in 725 seconds: 68.965515 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2