== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:54:19 (1786737259) start mds service on oleg211-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg211-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 143 (time 1786737375.96 total 10.04 last 14.25) - open/close 285 (time 1786737385.98 total 20.06 last 14.17) - open/close 519 (time 1786737396.95 total 31.03 last 21.33) - open/close 666 (time 1786737407.46 total 41.53 last 13.99) - open/close 851 (time 1786737417.46 total 51.53 last 18.49) - open/close 1039 (time 1786737428.17 total 62.24 last 17.56) - open/close 1153 (time 1786737438.17 total 72.24 last 11.39) - open/close 1413 (time 1786737448.62 total 82.69 last 24.89) - open/close 1651 (time 1786737458.92 total 92.99 last 23.10) - open/close 1904 (time 1786737469.44 total 103.51 last 24.06) - open/close 2186 (time 1786737479.44 total 113.51 last 28.19) - open/close 2315 (time 1786737490.47 total 124.55 last 11.69) - open/close 2550 (time 1786737500.48 total 134.55 last 23.49) - open/close 2776 (time 1786737510.48 total 144.56 last 22.59) - open/close 3015 (time 1786737520.67 total 154.74 last 23.46) - open/close 3251 (time 1786737530.68 total 164.75 last 23.58) - open/close 3461 (time 1786737540.98 total 175.05 last 20.38) - open/close 3729 (time 1786737551.01 total 185.08 last 26.73) - open/close 3878 (time 1786737561.59 total 195.66 last 14.08) - open/close 4137 (time 1786737572.66 total 206.74 last 23.39) - open/close 4301 (time 1786737582.69 total 216.76 last 16.37) - open/close 4464 (time 1786737593.01 total 227.09 last 15.78) - open/close 4667 (time 1786737603.24 total 237.32 last 19.84) - open/close 4832 (time 1786737613.76 total 247.83 last 15.70) - open/close 4976 (time 1786737623.92 total 257.99 last 14.17) - open/close 5231 (time 1786737633.92 total 267.99 last 25.49) - open/close 5454 (time 1786737644.57 total 278.64 last 20.94) - open/close 5639 (time 1786737655.22 total 289.29 last 17.37) - open/close 5821 (time 1786737665.49 total 299.57 last 17.71) - open/close 6113 (time 1786737675.52 total 309.60 last 29.11) - open/close 6321 (time 1786737685.55 total 319.62 last 20.75) - open/close 6543 (time 1786737696.93 total 331.00 last 19.51) - open/close 6691 (time 1786737707.11 total 341.18 last 14.54) - open/close 6995 (time 1786737718.18 total 352.25 last 27.45) - open/close 7256 (time 1786737728.19 total 362.26 last 26.08) - open/close 7472 (time 1786737738.20 total 372.27 last 21.58) - open/close 7650 (time 1786737748.64 total 382.71 last 17.04) - open/close 7849 (time 1786737758.88 total 392.95 last 19.44) - open/close 8053 (time 1786737769.72 total 403.79 last 18.82) - open/close 8153 (time 1786737782.12 total 416.19 last 8.06) - open/close 8395 (time 1786737792.86 total 426.93 last 22.54) - open/close 8633 (time 1786737805.94 total 440.01 last 18.20) - open/close 8862 (time 1786737816.28 total 450.35 last 22.14) - open/close 9107 (time 1786737826.49 total 460.56 last 24.00) - open/close 9290 (time 1786737836.97 total 471.04 last 17.47) - open/close 9419 (time 1786737846.97 total 481.04 last 12.89) - open/close 9708 (time 1786737856.98 total 491.06 last 28.86) - open/close 9982 (time 1786737866.99 total 501.06 last 27.38) - open/close 10221 (time 1786737877.00 total 511.07 last 23.89) - open/close 10415 (time 1786737887.00 total 521.08 last 19.39) - open/close 10652 (time 1786737897.01 total 531.08 last 23.69) - open/close 10843 (time 1786737907.01 total 541.08 last 19.10) - open/close 10999 (time 1786737917.01 total 551.08 last 15.60) - open/close 11274 (time 1786737927.47 total 561.54 last 26.29) - open/close 11541 (time 1786737938.29 total 572.37 last 24.67) - open/close 11639 (time 1786737948.29 total 582.37 last 9.80) - open/close 11878 (time 1786737958.45 total 592.53 last 23.52) - open/close 12100 (time 1786737969.15 total 603.22 last 20.76) - open/close 12376 (time 1786737980.19 total 614.26 last 25.00) - open/close 12596 (time 1786737991.85 total 625.92 last 18.86) - open/close 12868 (time 1786738001.86 total 635.93 last 27.19) - open/close 13177 (time 1786738011.86 total 645.93 last 30.89) - open/close 13391 (time 1786738022.88 total 656.96 last 19.41) - open/close 13583 (time 1786738033.58 total 667.66 last 17.95) - open/close 13685 (time 1786738044.00 total 678.07 last 9.79) - open/close 13830 (time 1786738054.62 total 688.70 last 13.64) - open/close 13997 (time 1786738065.61 total 699.68 last 15.21) - open/close 14278 (time 1786738075.62 total 709.69 last 28.06) - open/close 14536 (time 1786738085.95 total 720.02 last 24.98) - open/close 14814 (time 1786738096.36 total 730.43 last 26.70) - open/close 15028 (time 1786738108.27 total 742.34 last 17.97) - open/close 15156 (time 1786738118.30 total 752.38 last 12.76) - open/close 15437 (time 1786738128.55 total 762.62 last 27.44) - open/close 15606 (time 1786738140.20 total 774.27 last 14.50) - open/close 15863 (time 1786738150.21 total 784.28 last 25.68) - open/close 16021 (time 1786738160.80 total 794.87 last 14.92) - open/close 16096 (time 1786738170.82 total 804.89 last 7.49) - open/close 16236 (time 1786738181.80 total 815.87 last 12.75) - open/close 16496 (time 1786738192.25 total 826.32 last 24.87) - open/close 16725 (time 1786738202.27 total 836.35 last 22.85) - open/close 16950 (time 1786738212.28 total 846.36 last 22.48) - open/close 17073 (time 1786738222.30 total 856.38 last 12.27) - open/close 17273 (time 1786738232.75 total 866.82 last 19.14) - open/close 17433 (time 1786738242.75 total 876.83 last 15.99) - open/close 17606 (time 1786738253.60 total 887.67 last 15.95) - open/close 17873 (time 1786738264.45 total 898.52 last 24.60) - open/close 18003 (time 1786738274.91 total 908.98 last 12.43) - open/close 18280 (time 1786738284.91 total 918.98 last 27.69) - open/close 18562 (time 1786738294.91 total 928.98 last 28.20) - open/close 18746 (time 1786738304.92 total 938.99 last 18.39) - open/close 18975 (time 1786738315.24 total 949.31 last 22.18) - open/close 19154 (time 1786738326.19 total 960.26 last 16.35) - open/close 19376 (time 1786738336.57 total 970.64 last 21.38) - open/close 19558 (time 1786738347.11 total 981.18 last 17.27) - open/close 20000 (time 1786738356.76 total 990.83 last 45.81) - open/close 20945 (time 1786738366.76 total 1000.84 last 94.46) - open/close 21804 (time 1786738376.76 total 1010.84 last 85.89) - open/close 22814 (time 1786738386.77 total 1020.84 last 100.96) - open/close 23874 (time 1786738396.77 total 1030.84 last 105.99) - open/close 24866 (time 1786738406.78 total 1040.85 last 99.08) - open/close 25875 (time 1786738416.79 total 1050.86 last 100.83) - open/close 26691 (time 1786738426.79 total 1060.86 last 81.58) - open/close 27648 (time 1786738436.79 total 1070.87 last 95.67) - open/close 28394 (time 1786738446.80 total 1080.87 last 74.58) - open/close 29392 (time 1786738456.80 total 1090.88 last 99.72) - open/close 30000 (time 1786738463.62 total 1097.69 last 89.17) - open/close 30912 (time 1786738473.63 total 1107.70 last 91.14) - open/close 31579 (time 1786738483.64 total 1117.71 last 66.61) - open/close 32586 (time 1786738493.65 total 1127.72 last 100.64) - open/close 33560 (time 1786738503.67 total 1137.74 last 97.22) - open/close 34379 (time 1786738513.67 total 1147.74 last 81.88) - open/close 35108 (time 1786738523.69 total 1157.76 last 72.77) - open/close 35759 (time 1786738533.70 total 1167.77 last 65.02) - open/close 36650 (time 1786738543.70 total 1177.77 last 89.09) - open/close 37546 (time 1786738553.71 total 1187.78 last 89.53) - open/close 38516 (time 1786738563.71 total 1197.79 last 96.92) - open/close 39398 (time 1786738573.74 total 1207.82 last 87.94) - open/close 40000 (time 1786738579.83 total 1213.90 last 98.92) - open/close 41088 (time 1786738589.84 total 1223.91 last 108.72) - open/close 42120 (time 1786738599.84 total 1233.91 last 103.16) - open/close 43065 (time 1786738609.85 total 1243.92 last 94.46) - open/close 43935 (time 1786738619.85 total 1253.92 last 86.96) - open/close 44788 (time 1786738629.86 total 1263.93 last 85.22) - open/close 45613 (time 1786738639.87 total 1273.94 last 82.44) - open/close 46407 (time 1786738649.87 total 1283.95 last 79.33) - open/close 47311 (time 1786738659.88 total 1293.95 last 90.34) - open/close 48285 (time 1786738669.88 total 1303.95 last 97.40) - open/close 49312 (time 1786738679.89 total 1313.96 last 102.65) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1317.20 seconds: 37.68 ops/second - unlinked 0 (time 1786738687 ; total 0 ; last 0) - unlinked 10000 (time 1786738797 ; total 110 ; last 110) - unlinked 20000 (time 1786738895 ; total 208 ; last 98) - unlinked 30000 (time 1786738965 ; total 278 ; last 70) - unlinked 40000 (time 1786739026 ; total 339 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 396 seconds: 125.330811 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2