== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:53:08 (1785315188) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 110 (time 1785315300.21 total 10.47 last 10.51) - open/close 203 (time 1785315310.22 total 20.47 last 9.30) - open/close 410 (time 1785315321.02 total 31.28 last 19.16) - open/close 558 (time 1785315331.03 total 41.29 last 14.78) - open/close 803 (time 1785315341.06 total 51.32 last 24.43) - open/close 941 (time 1785315352.02 total 62.28 last 12.59) - open/close 1105 (time 1785315362.18 total 72.44 last 16.14) - open/close 1384 (time 1785315372.62 total 82.88 last 26.72) - open/close 1647 (time 1785315383.62 total 93.87 last 23.93) - open/close 1861 (time 1785315394.23 total 104.49 last 20.15) - open/close 2044 (time 1785315404.93 total 115.19 last 17.10) - open/close 2179 (time 1785315414.94 total 125.19 last 13.50) - open/close 2337 (time 1785315425.27 total 135.52 last 15.29) - open/close 2437 (time 1785315436.91 total 147.16 last 8.59) - open/close 2595 (time 1785315447.72 total 157.98 last 14.61) - open/close 2765 (time 1785315458.61 total 168.86 last 15.62) - open/close 2935 (time 1785315469.20 total 179.45 last 16.05) - open/close 3015 (time 1785315480.30 total 190.56 last 7.20) - open/close 3124 (time 1785315490.69 total 200.94 last 10.50) - open/close 3301 (time 1785315501.43 total 211.69 last 16.47) - open/close 3375 (time 1785315511.73 total 221.98 last 7.19) - open/close 3492 (time 1785315521.73 total 231.99 last 11.69) - open/close 3591 (time 1785315531.75 total 242.01 last 9.88) - open/close 3737 (time 1785315542.51 total 252.76 last 13.57) - open/close 3866 (time 1785315553.41 total 263.67 last 11.83) - open/close 4048 (time 1785315563.67 total 273.92 last 17.75) - open/close 4221 (time 1785315574.53 total 284.79 last 15.92) - open/close 4404 (time 1785315584.55 total 294.80 last 18.27) - open/close 4479 (time 1785315594.57 total 304.83 last 7.48) - open/close 4642 (time 1785315604.74 total 315.00 last 16.03) - open/close 4830 (time 1785315614.74 total 325.00 last 18.80) - open/close 4986 (time 1785315624.75 total 335.00 last 15.59) - open/close 5119 (time 1785315635.40 total 345.65 last 12.49) - open/close 5307 (time 1785315645.43 total 355.69 last 18.74) - open/close 5536 (time 1785315656.19 total 366.44 last 21.29) - open/close 5606 (time 1785315667.14 total 377.40 last 6.39) - open/close 5838 (time 1785315677.16 total 387.41 last 23.16) - open/close 6046 (time 1785315687.62 total 397.88 last 19.88) - open/close 6243 (time 1785315698.69 total 408.94 last 17.81) - open/close 6522 (time 1785315708.79 total 419.05 last 27.61) - open/close 6643 (time 1785315718.93 total 429.19 last 11.93) - open/close 6808 (time 1785315729.97 total 440.23 last 14.94) - open/close 6929 (time 1785315739.99 total 450.24 last 12.08) - open/close 7100 (time 1785315750.52 total 460.78 last 16.23) - open/close 7193 (time 1785315760.77 total 471.02 last 9.08) - open/close 7371 (time 1785315770.78 total 481.04 last 17.77) - open/close 7476 (time 1785315781.65 total 491.91 last 9.66) - open/close 7683 (time 1785315791.67 total 501.92 last 20.66) - open/close 7900 (time 1785315801.70 total 511.95 last 21.63) - open/close 8022 (time 1785315813.19 total 523.44 last 10.62) - open/close 8229 (time 1785315823.81 total 534.06 last 19.48) - open/close 8413 (time 1785315834.10 total 544.36 last 17.87) - open/close 8565 (time 1785315844.99 total 555.24 last 13.97) - open/close 8728 (time 1785315855.23 total 565.48 last 15.92) - open/close 8809 (time 1785315865.47 total 575.73 last 7.90) - open/close 8916 (time 1785315876.18 total 586.43 last 10.00) - open/close 9015 (time 1785315886.93 total 597.18 last 9.21) - open/close 9153 (time 1785315896.94 total 607.19 last 13.79) - open/close 9287 (time 1785315908.73 total 618.98 last 11.36) - open/close 9523 (time 1785315918.87 total 629.12 last 23.28) - open/close 9721 (time 1785315928.88 total 639.14 last 19.77) - open/close 9866 (time 1785315938.95 total 649.20 last 14.41) - open/close 9970 (time 1785315949.27 total 659.52 last 10.08) - open/close 10000 (time 1785315951.31 total 661.56 last 14.69) - open/close 10132 (time 1785315961.32 total 671.57 last 13.18) - open/close 10241 (time 1785315971.84 total 682.09 last 10.36) - open/close 10426 (time 1785315982.79 total 693.04 last 16.89) - open/close 10548 (time 1785315993.02 total 703.28 last 11.92) - open/close 10599 (time 1785316003.43 total 713.68 last 4.90) - open/close 10797 (time 1785316013.46 total 723.71 last 19.74) - open/close 11002 (time 1785316023.68 total 733.93 last 20.05) - open/close 11226 (time 1785316034.17 total 744.43 last 21.35) - open/close 11404 (time 1785316044.92 total 755.17 last 16.57) - open/close 11675 (time 1785316055.54 total 765.79 last 25.51) - open/close 11799 (time 1785316065.54 total 775.80 last 12.40) - open/close 11992 (time 1785316076.09 total 786.35 last 18.29) - open/close 12163 (time 1785316086.10 total 796.36 last 17.08) - open/close 12336 (time 1785316096.83 total 807.09 last 16.13) - open/close 12511 (time 1785316106.90 total 817.15 last 17.39) - open/close 12638 (time 1785316117.51 total 827.76 last 11.97) - open/close 12764 (time 1785316127.52 total 837.77 last 12.59) - open/close 13028 (time 1785316138.36 total 848.61 last 24.36) - open/close 13160 (time 1785316149.17 total 859.42 last 12.21) - open/close 13321 (time 1785316159.41 total 869.67 last 15.71) - open/close 13532 (time 1785316170.33 total 880.58 last 19.34) - open/close 13792 (time 1785316180.34 total 890.59 last 25.97) - open/close 13975 (time 1785316190.35 total 900.60 last 18.28) - open/close 14152 (time 1785316201.22 total 911.47 last 16.29) - open/close 14321 (time 1785316211.91 total 922.17 last 15.80) - open/close 14524 (time 1785316221.92 total 932.18 last 20.28) - open/close 14677 (time 1785316231.93 total 942.18 last 15.29) - open/close 14877 (time 1785316241.96 total 952.21 last 19.94) - open/close 15024 (time 1785316255.05 total 965.30 last 11.23) - open/close 15143 (time 1785316265.07 total 975.33 last 11.87) - open/close 15324 (time 1785316275.08 total 985.33 last 18.09) - open/close 15530 (time 1785316285.26 total 995.51 last 20.23) - open/close 15666 (time 1785316295.35 total 1005.61 last 13.47) - open/close 15848 (time 1785316305.99 total 1016.25 last 17.11) - open/close 15975 (time 1785316317.56 total 1027.81 last 10.98) - open/close 16230 (time 1785316327.56 total 1037.81 last 25.50) - open/close 16435 (time 1785316338.16 total 1048.41 last 19.34) - open/close 16599 (time 1785316348.67 total 1058.92 last 15.60) - open/close 16768 (time 1785316358.68 total 1068.94 last 16.88) - open/close 16861 (time 1785316369.49 total 1079.74 last 8.60) - open/close 17015 (time 1785316379.51 total 1089.76 last 15.37) - open/close 17160 (time 1785316389.53 total 1099.79 last 14.46) - open/close 17335 (time 1785316399.54 total 1109.79 last 17.49) - open/close 17495 (time 1785316409.61 total 1119.86 last 15.89) - open/close 17680 (time 1785316419.61 total 1129.87 last 18.49) - open/close 18402 (time 1785316429.64 total 1139.89 last 72.00) - open/close 19204 (time 1785316439.65 total 1149.90 last 80.15) - open/close 19979 (time 1785316449.65 total 1159.90 last 77.47) - open/close 20686 (time 1785316459.66 total 1169.91 last 70.64) - open/close 21486 (time 1785316469.66 total 1179.92 last 79.95) - open/close 22170 (time 1785316479.69 total 1189.94 last 68.24) - open/close 22894 (time 1785316489.73 total 1199.98 last 72.12) - open/close 23632 (time 1785316499.73 total 1209.99 last 73.75) - open/close 24554 (time 1785316509.75 total 1220.00 last 92.09) - open/close 25426 (time 1785316519.76 total 1230.01 last 87.11) - open/close 26215 (time 1785316529.76 total 1240.02 last 78.86) - open/close 27351 (time 1785316539.76 total 1250.02 last 113.57) - open/close 28178 (time 1785316549.78 total 1260.04 last 82.56) - open/close 29009 (time 1785316559.78 total 1270.04 last 83.08) - open/close 29909 (time 1785316569.79 total 1280.04 last 89.96) - open/close 30630 (time 1785316579.80 total 1290.05 last 72.05) - open/close 31254 (time 1785316589.81 total 1300.06 last 62.33) - open/close 31828 (time 1785316599.82 total 1310.07 last 57.34) - open/close 32598 (time 1785316609.82 total 1320.07 last 76.98) - open/close 33542 (time 1785316619.86 total 1330.12 last 94.00) - open/close 34125 (time 1785316629.88 total 1340.14 last 58.17) - open/close 34782 (time 1785316639.93 total 1350.18 last 65.43) - open/close 35250 (time 1785316649.99 total 1360.24 last 46.52) - open/close 35732 (time 1785316660.00 total 1370.25 last 48.14) - open/close 36563 (time 1785316670.01 total 1380.26 last 83.04) - open/close 37332 (time 1785316680.01 total 1390.26 last 76.89) - open/close 38137 (time 1785316690.04 total 1400.29 last 80.26) - open/close 38880 (time 1785316700.05 total 1410.30 last 74.22) - open/close 39873 (time 1785316710.05 total 1420.30 last 99.30) - open/close 40970 (time 1785316720.05 total 1430.30 last 109.68) - open/close 41817 (time 1785316730.05 total 1440.31 last 84.67) - open/close 42619 (time 1785316740.05 total 1450.31 last 80.20) - open/close 43462 (time 1785316750.07 total 1460.32 last 84.18) - open/close 44295 (time 1785316760.07 total 1470.32 last 83.30) - open/close 45072 (time 1785316770.07 total 1480.33 last 77.67) - open/close 46022 (time 1785316780.07 total 1490.33 last 94.97) - open/close 46910 (time 1785316790.08 total 1500.33 last 88.75) - open/close 47987 (time 1785316800.08 total 1510.34 last 107.66) - open/close 48880 (time 1785316810.09 total 1520.34 last 89.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1529.21 seconds: 32.46 ops/second - unlinked 0 (time 1785316823 ; total 0 ; last 0) - unlinked 10000 (time 1785316956 ; total 133 ; last 133) - unlinked 20000 (time 1785317076 ; total 253 ; last 120) - unlinked 30000 (time 1785317206 ; total 383 ; last 130) - unlinked 40000 (time 1785317345 ; total 522 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 643 seconds: 77.186623 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded. pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2