== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:09:16 (1786748956) start mds service on oleg124-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg124-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 210 (time 1786749066.85 total 11.02 last 19.05) - open/close 346 (time 1786749076.86 total 21.04 last 13.59) - open/close 522 (time 1786749087.06 total 31.24 last 17.25) - open/close 764 (time 1786749097.62 total 41.80 last 22.91) - open/close 1004 (time 1786749107.63 total 51.81 last 23.98) - open/close 1223 (time 1786749117.97 total 62.15 last 21.17) - open/close 1368 (time 1786749128.40 total 72.58 last 13.91) - open/close 1546 (time 1786749138.98 total 83.15 last 16.83) - open/close 1716 (time 1786749149.59 total 93.76 last 16.02) - open/close 1972 (time 1786749159.96 total 104.14 last 24.68) - open/close 2257 (time 1786749169.96 total 114.14 last 28.50) - open/close 2474 (time 1786749179.98 total 124.16 last 21.65) - open/close 2715 (time 1786749189.99 total 134.17 last 24.08) - open/close 2960 (time 1786749200.00 total 144.18 last 24.47) - open/close 3137 (time 1786749211.65 total 155.83 last 15.20) - open/close 3327 (time 1786749221.67 total 165.85 last 18.96) - open/close 3478 (time 1786749232.16 total 176.34 last 14.39) - open/close 3717 (time 1786749242.18 total 186.35 last 23.87) - open/close 3867 (time 1786749252.50 total 196.68 last 14.53) - open/close 4046 (time 1786749262.93 total 207.11 last 17.16) - open/close 4308 (time 1786749273.05 total 217.23 last 25.89) - open/close 4517 (time 1786749283.86 total 228.04 last 19.32) - open/close 4722 (time 1786749294.69 total 238.87 last 18.94) - open/close 4787 (time 1786749304.71 total 248.88 last 6.49) - open/close 5082 (time 1786749315.61 total 259.79 last 27.06) - open/close 5304 (time 1786749325.62 total 269.80 last 22.18) - open/close 5527 (time 1786749338.26 total 282.43 last 17.65) - open/close 5787 (time 1786749349.21 total 293.38 last 23.74) - open/close 6038 (time 1786749360.11 total 304.29 last 23.01) - open/close 6254 (time 1786749370.64 total 314.82 last 20.51) - open/close 6495 (time 1786749381.47 total 325.65 last 22.25) - open/close 6841 (time 1786749392.14 total 336.32 last 32.43) - open/close 7026 (time 1786749402.15 total 346.32 last 18.50) - open/close 7148 (time 1786749412.15 total 356.33 last 12.19) - open/close 7419 (time 1786749422.16 total 366.34 last 27.08) - open/close 7492 (time 1786749432.86 total 377.04 last 6.83) - open/close 7712 (time 1786749442.86 total 387.04 last 22.00) - open/close 7997 (time 1786749453.82 total 398.00 last 26.00) - open/close 8282 (time 1786749464.59 total 408.77 last 26.46) - open/close 8443 (time 1786749475.08 total 419.26 last 15.34) - open/close 8611 (time 1786749485.09 total 429.27 last 16.79) - open/close 8871 (time 1786749495.31 total 439.49 last 25.43) - open/close 9052 (time 1786749505.33 total 449.51 last 18.06) - open/close 9254 (time 1786749515.37 total 459.55 last 20.12) - open/close 9364 (time 1786749525.92 total 470.09 last 10.43) - open/close 9604 (time 1786749536.20 total 480.38 last 23.33) - open/close 9883 (time 1786749547.22 total 491.40 last 25.33) - open/close 10000 (time 1786749554.23 total 498.41 last 16.69) - open/close 10295 (time 1786749565.20 total 509.38 last 26.88) - open/close 10485 (time 1786749575.40 total 519.57 last 18.64) - open/close 10734 (time 1786749585.69 total 529.87 last 24.19) - open/close 11003 (time 1786749595.69 total 539.87 last 26.89) - open/close 11288 (time 1786749606.17 total 550.35 last 27.20) - open/close 11545 (time 1786749616.46 total 560.64 last 24.97) - open/close 11658 (time 1786749626.77 total 570.95 last 10.96) - open/close 11924 (time 1786749636.78 total 580.96 last 26.58) - open/close 12200 (time 1786749646.99 total 591.17 last 27.02) - open/close 12296 (time 1786749657.51 total 601.69 last 9.13) - open/close 12522 (time 1786749667.68 total 611.86 last 22.21) - open/close 12703 (time 1786749677.70 total 621.87 last 18.07) - open/close 12939 (time 1786749687.86 total 632.04 last 23.22) - open/close 13193 (time 1786749697.89 total 642.07 last 25.33) - open/close 13448 (time 1786749708.18 total 652.36 last 24.77) - open/close 13709 (time 1786749720.12 total 664.30 last 21.87) - open/close 13930 (time 1786749730.12 total 674.30 last 22.09) - open/close 14217 (time 1786749740.38 total 684.56 last 27.98) - open/close 14397 (time 1786749750.87 total 695.05 last 17.15) - open/close 14659 (time 1786749760.88 total 705.06 last 26.18) - open/close 14867 (time 1786749771.30 total 715.48 last 19.97) - open/close 15055 (time 1786749781.30 total 725.48 last 18.79) - open/close 15281 (time 1786749791.32 total 735.50 last 22.56) - open/close 15528 (time 1786749801.32 total 745.50 last 24.70) - open/close 15704 (time 1786749813.33 total 757.51 last 14.64) - open/close 15955 (time 1786749823.92 total 768.10 last 23.70) - open/close 16243 (time 1786749834.57 total 778.75 last 27.05) - open/close 16511 (time 1786749844.58 total 788.76 last 26.78) - open/close 16781 (time 1786749855.59 total 799.77 last 24.51) - open/close 16878 (time 1786749865.94 total 810.12 last 9.37) - open/close 17048 (time 1786749875.95 total 820.13 last 16.99) - open/close 17298 (time 1786749886.55 total 830.73 last 23.59) - open/close 17435 (time 1786749897.04 total 841.22 last 13.05) - open/close 17705 (time 1786749910.05 total 854.22 last 20.76) - open/close 17959 (time 1786749920.18 total 864.35 last 25.07) - open/close 18167 (time 1786749930.64 total 874.82 last 19.88) - open/close 18408 (time 1786749940.64 total 884.82 last 24.09) - open/close 18607 (time 1786749952.42 total 896.60 last 16.90) - open/close 18892 (time 1786749963.04 total 907.22 last 26.84) - open/close 19123 (time 1786749973.45 total 917.63 last 22.18) - open/close 19372 (time 1786749983.46 total 927.64 last 24.88) - open/close 19646 (time 1786749994.33 total 938.51 last 25.21) - open/close 19952 (time 1786750004.86 total 949.04 last 29.07) - open/close 20146 (time 1786750015.33 total 959.51 last 18.54) - open/close 20364 (time 1786750025.96 total 970.14 last 20.50) - open/close 21216 (time 1786750035.97 total 980.15 last 85.11) - open/close 22267 (time 1786750045.97 total 990.15 last 105.09) - open/close 23339 (time 1786750055.98 total 1000.16 last 107.14) - open/close 24207 (time 1786750065.98 total 1010.16 last 86.77) - open/close 25083 (time 1786750075.99 total 1020.17 last 87.55) - open/close 25747 (time 1786750086.00 total 1030.17 last 66.36) - open/close 26441 (time 1786750096.00 total 1040.18 last 69.35) - open/close 27306 (time 1786750106.00 total 1050.18 last 86.49) - open/close 28103 (time 1786750116.00 total 1060.18 last 79.69) - open/close 28901 (time 1786750126.01 total 1070.19 last 79.78) - open/close 29865 (time 1786750136.01 total 1080.19 last 96.38) - open/close 30794 (time 1786750146.02 total 1090.20 last 92.81) - open/close 31815 (time 1786750156.02 total 1100.20 last 102.07) - open/close 32688 (time 1786750166.03 total 1110.21 last 87.26) - open/close 33808 (time 1786750176.04 total 1120.21 last 111.90) - open/close 34926 (time 1786750186.04 total 1130.22 last 111.79) - open/close 35640 (time 1786750196.06 total 1140.23 last 71.27) - open/close 36523 (time 1786750206.06 total 1150.24 last 88.27) - open/close 37552 (time 1786750216.06 total 1160.24 last 102.89) - open/close 38333 (time 1786750226.07 total 1170.25 last 77.99) - open/close 39103 (time 1786750236.08 total 1180.26 last 76.94) - open/close 39992 (time 1786750246.08 total 1190.26 last 88.88) - open/close 40734 (time 1786750256.09 total 1200.27 last 74.16) - open/close 41735 (time 1786750266.10 total 1210.27 last 100.05) - open/close 42515 (time 1786750276.10 total 1220.28 last 77.98) - open/close 43384 (time 1786750286.10 total 1230.28 last 86.90) - open/close 44164 (time 1786750296.11 total 1240.29 last 77.92) - open/close 45047 (time 1786750306.11 total 1250.29 last 88.30) - open/close 46007 (time 1786750316.12 total 1260.29 last 95.94) - open/close 46931 (time 1786750326.12 total 1270.30 last 92.37) - open/close 47900 (time 1786750336.13 total 1280.31 last 96.80) - open/close 48583 (time 1786750346.13 total 1290.31 last 68.28) - open/close 49300 (time 1786750356.13 total 1300.31 last 71.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1303.99 seconds: 38.06 ops/second - unlinked 0 (time 1786750364 ; total 0 ; last 0) - unlinked 10000 (time 1786750494 ; total 130 ; last 130) - unlinked 20000 (time 1786750609 ; total 245 ; last 115) - unlinked 30000 (time 1786750675 ; total 311 ; last 66) - unlinked 40000 (time 1786750746 ; total 382 ; last 71) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 442 seconds: 112.287331 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: modules unloaded. pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2