== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:00:41 (1787587241) start mds service on oleg431-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg431-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 96 (time 1787587358.73 total 10.19 last 9.42) - open/close 285 (time 1787587368.75 total 20.21 last 18.87) - open/close 483 (time 1787587378.76 total 30.21 last 19.79) - open/close 567 (time 1787587388.94 total 40.39 last 8.25) - open/close 739 (time 1787587398.95 total 50.41 last 17.17) - open/close 901 (time 1787587408.97 total 60.43 last 16.18) - open/close 1097 (time 1787587418.98 total 70.44 last 19.57) - open/close 1306 (time 1787587429.01 total 80.47 last 20.84) - open/close 1589 (time 1787587439.57 total 91.02 last 26.81) - open/close 1777 (time 1787587450.38 total 101.84 last 17.38) - open/close 1876 (time 1787587460.48 total 111.94 last 9.81) - open/close 2098 (time 1787587471.20 total 122.66 last 20.71) - open/close 2254 (time 1787587481.21 total 132.67 last 15.58) - open/close 2447 (time 1787587492.00 total 143.45 last 17.89) - open/close 2679 (time 1787587502.89 total 154.34 last 21.31) - open/close 2941 (time 1787587512.90 total 164.35 last 26.17) - open/close 3063 (time 1787587522.96 total 174.42 last 12.12) - open/close 3270 (time 1787587533.21 total 184.67 last 20.20) - open/close 3464 (time 1787587543.22 total 194.68 last 19.38) - open/close 3577 (time 1787587554.07 total 205.53 last 10.41) - open/close 3649 (time 1787587564.08 total 215.54 last 7.20) - open/close 3775 (time 1787587574.47 total 225.93 last 12.12) - open/close 3953 (time 1787587584.48 total 235.94 last 17.78) - open/close 4127 (time 1787587595.35 total 246.80 last 16.02) - open/close 4192 (time 1787587605.36 total 256.82 last 6.49) - open/close 4266 (time 1787587615.37 total 266.83 last 7.39) - open/close 4442 (time 1787587626.06 total 277.52 last 16.47) - open/close 4509 (time 1787587636.95 total 288.40 last 6.15) - open/close 4661 (time 1787587646.95 total 298.41 last 15.19) - open/close 4912 (time 1787587656.98 total 308.44 last 25.02) - open/close 4988 (time 1787587666.99 total 318.44 last 7.60) - open/close 5149 (time 1787587678.22 total 329.68 last 14.33) - open/close 5295 (time 1787587688.23 total 339.69 last 14.59) - open/close 5489 (time 1787587698.25 total 349.70 last 19.37) - open/close 5700 (time 1787587708.26 total 359.72 last 21.08) - open/close 5961 (time 1787587718.80 total 370.26 last 24.75) - open/close 6225 (time 1787587728.83 total 380.29 last 26.32) - open/close 6383 (time 1787587738.84 total 390.30 last 15.78) - open/close 6553 (time 1787587749.13 total 400.59 last 16.52) - open/close 6730 (time 1787587759.14 total 410.60 last 17.68) - open/close 6879 (time 1787587770.19 total 421.64 last 13.49) - open/close 6893 (time 1787587780.68 total 432.14 last 1.33) - open/close 7064 (time 1787587792.47 total 443.93 last 14.50) - open/close 7271 (time 1787587802.64 total 454.10 last 20.35) - open/close 7489 (time 1787587812.66 total 464.12 last 21.75) - open/close 7590 (time 1787587824.26 total 475.72 last 8.70) - open/close 7700 (time 1787587834.66 total 486.12 last 10.58) - open/close 7968 (time 1787587847.02 total 498.47 last 21.69) - open/close 8132 (time 1787587857.20 total 508.66 last 16.11) - open/close 8185 (time 1787587872.22 total 523.67 last 3.53) - open/close 8308 (time 1787587882.23 total 533.69 last 12.28) - open/close 8398 (time 1787587893.47 total 544.93 last 8.00) - open/close 8557 (time 1787587903.48 total 554.94 last 15.89) - open/close 8804 (time 1787587913.50 total 564.96 last 24.66) - open/close 8970 (time 1787587924.82 total 576.28 last 14.66) - open/close 9153 (time 1787587936.33 total 587.78 last 15.91) - open/close 9318 (time 1787587949.99 total 601.44 last 12.08) - open/close 9476 (time 1787587962.76 total 614.22 last 12.37) - open/close 9679 (time 1787587972.77 total 624.23 last 20.29) - open/close 9864 (time 1787587983.79 total 635.25 last 16.79) - open/close 10000 (time 1787587992.80 total 644.26 last 15.09) - open/close 10233 (time 1787588003.78 total 655.23 last 21.23) - open/close 10438 (time 1787588014.68 total 666.14 last 18.79) - open/close 10522 (time 1787588025.01 total 676.47 last 8.13) - open/close 10656 (time 1787588035.01 total 686.47 last 13.40) - open/close 10782 (time 1787588045.02 total 696.48 last 12.59) - open/close 11026 (time 1787588055.04 total 706.50 last 24.36) - open/close 11138 (time 1787588066.00 total 717.46 last 10.22) - open/close 11250 (time 1787588076.76 total 728.22 last 10.40) - open/close 11363 (time 1787588087.58 total 739.04 last 10.45) - open/close 11540 (time 1787588097.59 total 749.05 last 17.68) - open/close 11635 (time 1787588109.90 total 761.36 last 7.72) - open/close 11779 (time 1787588119.92 total 771.38 last 14.37) - open/close 12005 (time 1787588129.93 total 781.39 last 22.58) - open/close 12223 (time 1787588139.96 total 791.42 last 21.73) - open/close 12494 (time 1787588150.42 total 801.87 last 25.91) - open/close 12732 (time 1787588161.49 total 812.94 last 21.50) - open/close 12850 (time 1787588171.50 total 822.96 last 11.78) - open/close 12977 (time 1787588181.74 total 833.20 last 12.41) - open/close 13141 (time 1787588192.02 total 843.48 last 15.95) - open/close 13165 (time 1787588202.05 total 853.51 last 2.39) - open/close 13309 (time 1787588212.49 total 863.95 last 13.80) - open/close 13386 (time 1787588222.82 total 874.28 last 7.46) - open/close 13506 (time 1787588232.85 total 884.31 last 11.96) - open/close 13567 (time 1787588243.77 total 895.23 last 5.59) - open/close 13646 (time 1787588253.86 total 905.32 last 7.83) - open/close 13897 (time 1787588263.86 total 915.32 last 25.09) - open/close 14104 (time 1787588275.50 total 926.95 last 17.80) - open/close 14218 (time 1787588285.72 total 937.17 last 11.15) - open/close 14274 (time 1787588296.41 total 947.87 last 5.24) - open/close 14375 (time 1787588307.14 total 958.60 last 9.41) - open/close 14405 (time 1787588317.26 total 968.72 last 2.97) - open/close 14453 (time 1787588327.88 total 979.34 last 4.52) - open/close 14592 (time 1787588338.64 total 990.10 last 12.92) - open/close 14760 (time 1787588350.73 total 1002.19 last 13.90) - open/close 14834 (time 1787588360.75 total 1012.20 last 7.39) - open/close 14917 (time 1787588372.43 total 1023.89 last 7.10) - open/close 15044 (time 1787588382.48 total 1033.94 last 12.64) - open/close 15143 (time 1787588393.36 total 1044.82 last 9.10) - open/close 15243 (time 1787588403.37 total 1054.83 last 9.99) - open/close 15434 (time 1787588413.40 total 1064.86 last 19.05) - open/close 16051 (time 1787588423.40 total 1074.86 last 61.66) - open/close 16755 (time 1787588433.41 total 1084.86 last 70.39) - open/close 17459 (time 1787588443.41 total 1094.87 last 70.34) - open/close 18216 (time 1787588453.42 total 1104.88 last 75.66) - open/close 18894 (time 1787588463.43 total 1114.88 last 67.75) - open/close 19637 (time 1787588473.43 total 1124.89 last 74.26) - open/close 20000 (time 1787588477.59 total 1129.05 last 87.22) - open/close 20700 (time 1787588487.61 total 1139.07 last 69.89) - open/close 21469 (time 1787588497.61 total 1149.07 last 76.86) - open/close 22258 (time 1787588507.61 total 1159.07 last 78.89) - open/close 23032 (time 1787588517.63 total 1169.08 last 77.31) - open/close 23767 (time 1787588527.63 total 1179.09 last 73.47) - open/close 24547 (time 1787588537.64 total 1189.10 last 77.88) - open/close 25186 (time 1787588547.65 total 1199.11 last 63.86) - open/close 25968 (time 1787588557.67 total 1209.12 last 78.09) - open/close 26650 (time 1787588567.70 total 1219.16 last 67.97) - open/close 27277 (time 1787588577.71 total 1229.17 last 62.62) - open/close 27927 (time 1787588587.72 total 1239.17 last 64.98) - open/close 28706 (time 1787588597.72 total 1249.17 last 77.89) - open/close 29398 (time 1787588607.72 total 1259.18 last 69.19) - open/close 30000 (time 1787588617.12 total 1268.58 last 64.02) - open/close 30661 (time 1787588627.14 total 1278.60 last 65.97) - open/close 31307 (time 1787588637.14 total 1288.60 last 64.59) - open/close 32001 (time 1787588647.14 total 1298.60 last 69.40) - open/close 32698 (time 1787588657.14 total 1308.60 last 69.69) - open/close 33432 (time 1787588667.15 total 1318.60 last 73.38) - open/close 34137 (time 1787588677.15 total 1328.61 last 70.47) - open/close 34800 (time 1787588687.15 total 1338.61 last 66.29) - open/close 35463 (time 1787588697.17 total 1348.63 last 66.17) - open/close 36056 (time 1787588707.18 total 1358.63 last 59.28) - open/close 36705 (time 1787588717.18 total 1368.64 last 64.86) - open/close 37309 (time 1787588727.19 total 1378.65 last 60.35) - open/close 37921 (time 1787588737.20 total 1388.65 last 61.16) - open/close 38579 (time 1787588747.20 total 1398.66 last 65.78) - open/close 39281 (time 1787588757.23 total 1408.68 last 70.00) - open/close 39943 (time 1787588767.24 total 1418.69 last 66.13) - open/close 40659 (time 1787588777.24 total 1428.70 last 71.58) - open/close 41400 (time 1787588787.25 total 1438.70 last 74.05) - open/close 42036 (time 1787588797.27 total 1448.73 last 63.44) - open/close 42723 (time 1787588807.29 total 1458.74 last 68.59) - open/close 43452 (time 1787588817.29 total 1468.75 last 72.85) - open/close 44139 (time 1787588827.30 total 1478.76 last 68.64) - open/close 44841 (time 1787588837.31 total 1488.77 last 70.12) - open/close 45594 (time 1787588847.32 total 1498.78 last 75.26) - open/close 46346 (time 1787588857.32 total 1508.78 last 75.18) - open/close 47035 (time 1787588867.33 total 1518.79 last 68.84) - open/close 47768 (time 1787588877.34 total 1528.80 last 73.22) - open/close 48461 (time 1787588887.34 total 1538.80 last 69.29) - open/close 49183 (time 1787588897.35 total 1548.81 last 72.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1555.27 seconds: 31.91 ops/second - unlinked 0 (time 1787588908 ; total 0 ; last 0) - unlinked 10000 (time 1787589055 ; total 147 ; last 147) - unlinked 20000 (time 1787589201 ; total 293 ; last 146) - unlinked 30000 (time 1787589352 ; total 444 ; last 151) - unlinked 40000 (time 1787589498 ; total 590 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 733 seconds: 67.709412 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: modules unloaded. pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2