== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:10:15 (1786407015) start mds service on oleg249-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg249-server: detected 4 online CPUs by sysfs oleg249-server: MODOPTS_LIBCFS= oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg249-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg249-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg249-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg249-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1786407132.04 total 10.12 last 14.24) - open/close 370 (time 1786407142.21 total 20.29 last 22.21) - open/close 620 (time 1786407152.22 total 30.30 last 24.98) - open/close 921 (time 1786407162.23 total 40.31 last 30.06) - open/close 993 (time 1786407172.48 total 50.56 last 7.03) - open/close 1085 (time 1786407184.00 total 62.08 last 7.98) - open/close 1223 (time 1786407194.50 total 72.58 last 13.15) - open/close 1379 (time 1786407206.66 total 84.74 last 12.83) - open/close 1497 (time 1786407218.14 total 96.22 last 10.28) - open/close 1625 (time 1786407228.14 total 106.22 last 12.79) - open/close 1797 (time 1786407238.14 total 116.22 last 17.20) - open/close 1936 (time 1786407248.73 total 126.81 last 13.14) - open/close 2048 (time 1786407258.76 total 136.84 last 11.17) - open/close 2241 (time 1786407269.17 total 147.25 last 18.52) - open/close 2341 (time 1786407279.62 total 157.70 last 9.58) - open/close 2573 (time 1786407289.63 total 167.71 last 23.18) - open/close 2752 (time 1786407302.05 total 180.13 last 14.41) - open/close 2830 (time 1786407313.02 total 191.10 last 7.10) - open/close 3012 (time 1786407323.33 total 201.41 last 17.66) - open/close 3228 (time 1786407334.27 total 212.35 last 19.74) - open/close 3363 (time 1786407345.10 total 223.18 last 12.46) - open/close 3466 (time 1786407355.14 total 233.22 last 10.27) - open/close 3549 (time 1786407365.39 total 243.47 last 8.09) - open/close 3710 (time 1786407375.93 total 254.01 last 15.27) - open/close 3831 (time 1786407385.94 total 264.02 last 12.09) - open/close 3950 (time 1786407396.12 total 274.20 last 11.69) - open/close 4028 (time 1786407406.35 total 284.43 last 7.62) - open/close 4157 (time 1786407417.01 total 295.09 last 12.10) - open/close 4277 (time 1786407427.05 total 305.13 last 11.96) - open/close 4356 (time 1786407437.07 total 315.15 last 7.89) - open/close 4609 (time 1786407447.07 total 325.15 last 25.28) - open/close 4837 (time 1786407457.38 total 335.46 last 22.13) - open/close 4996 (time 1786407467.41 total 345.49 last 15.85) - open/close 5197 (time 1786407477.43 total 355.51 last 20.06) - open/close 5500 (time 1786407487.44 total 365.52 last 30.27) - open/close 5615 (time 1786407497.53 total 375.61 last 11.40) - open/close 5793 (time 1786407508.40 total 386.48 last 16.37) - open/close 5913 (time 1786407519.36 total 397.44 last 10.95) - open/close 6006 (time 1786407530.05 total 408.13 last 8.70) - open/close 6240 (time 1786407540.44 total 418.52 last 22.53) - open/close 6436 (time 1786407550.45 total 428.53 last 19.56) - open/close 6672 (time 1786407561.22 total 439.30 last 21.92) - open/close 6807 (time 1786407573.38 total 451.46 last 11.11) - open/close 7107 (time 1786407583.38 total 461.46 last 29.99) - open/close 7321 (time 1786407593.78 total 471.86 last 20.57) - open/close 7456 (time 1786407604.11 total 482.19 last 13.07) - open/close 7663 (time 1786407615.61 total 493.69 last 18.01) - open/close 7885 (time 1786407625.67 total 503.75 last 22.05) - open/close 8174 (time 1786407636.61 total 514.69 last 26.43) - open/close 8413 (time 1786407646.64 total 524.72 last 23.83) - open/close 8634 (time 1786407656.65 total 534.73 last 22.07) - open/close 8824 (time 1786407666.68 total 544.76 last 18.94) - open/close 8969 (time 1786407677.35 total 555.43 last 13.60) - open/close 9121 (time 1786407687.88 total 565.96 last 14.43) - open/close 9408 (time 1786407698.98 total 577.06 last 25.84) - open/close 9564 (time 1786407709.01 total 587.09 last 15.55) - open/close 9828 (time 1786407719.02 total 597.10 last 26.39) - open/close 9971 (time 1786407729.30 total 607.38 last 13.91) - open/close 10178 (time 1786407740.20 total 618.28 last 18.99) - open/close 10369 (time 1786407751.10 total 629.18 last 17.53) - open/close 10590 (time 1786407761.11 total 639.19 last 22.07) - open/close 10731 (time 1786407771.19 total 649.27 last 13.99) - open/close 10924 (time 1786407781.23 total 659.31 last 19.22) - open/close 11135 (time 1786407792.01 total 670.09 last 19.57) - open/close 11266 (time 1786407802.43 total 680.51 last 12.58) - open/close 11490 (time 1786407813.25 total 691.33 last 20.70) - open/close 11704 (time 1786407823.35 total 701.43 last 21.18) - open/close 11900 (time 1786407833.40 total 711.48 last 19.52) - open/close 12153 (time 1786407843.73 total 721.81 last 24.48) - open/close 12406 (time 1786407854.24 total 732.32 last 24.09) - open/close 12673 (time 1786407864.83 total 742.91 last 25.20) - open/close 12919 (time 1786407874.84 total 752.92 last 24.58) - open/close 13139 (time 1786407885.04 total 763.12 last 21.56) - open/close 13352 (time 1786407895.49 total 773.57 last 20.39) - open/close 13479 (time 1786407905.92 total 784.00 last 12.17) - open/close 13597 (time 1786407916.11 total 794.19 last 11.58) - open/close 13662 (time 1786407926.16 total 804.24 last 6.47) - open/close 13799 (time 1786407936.17 total 814.25 last 13.69) - open/close 13986 (time 1786407946.44 total 824.52 last 18.20) - open/close 14193 (time 1786407956.45 total 834.53 last 20.68) - open/close 14469 (time 1786407967.49 total 845.57 last 25.01) - open/close 14723 (time 1786407978.49 total 856.57 last 23.08) - open/close 14978 (time 1786407988.92 total 867.00 last 24.46) - open/close 15150 (time 1786407999.10 total 877.18 last 16.89) - open/close 15399 (time 1786408010.10 total 888.18 last 22.64) - open/close 15655 (time 1786408021.00 total 899.08 last 23.48) - open/close 15865 (time 1786408031.02 total 909.10 last 20.96) - open/close 16098 (time 1786408041.02 total 919.10 last 23.30) - open/close 16280 (time 1786408051.02 total 929.10 last 18.19) - open/close 16502 (time 1786408061.96 total 940.04 last 20.30) - open/close 16574 (time 1786408072.10 total 950.18 last 7.10) - open/close 16720 (time 1786408082.96 total 961.04 last 13.45) - open/close 16929 (time 1786408092.97 total 971.05 last 20.87) - open/close 17182 (time 1786408102.97 total 981.05 last 25.30) - open/close 17331 (time 1786408113.48 total 991.56 last 14.18) - open/close 17450 (time 1786408124.08 total 1002.16 last 11.23) - open/close 17574 (time 1786408134.10 total 1012.18 last 12.37) - open/close 17755 (time 1786408144.12 total 1022.20 last 18.07) - open/close 17851 (time 1786408154.18 total 1032.26 last 9.54) - open/close 18001 (time 1786408164.29 total 1042.37 last 14.84) - open/close 18397 (time 1786408174.29 total 1052.37 last 39.57) - open/close 19132 (time 1786408184.29 total 1062.37 last 73.50) - open/close 19926 (time 1786408194.30 total 1072.38 last 79.38) - open/close 20695 (time 1786408204.31 total 1082.39 last 76.81) - open/close 21516 (time 1786408214.31 total 1092.39 last 82.09) - open/close 22319 (time 1786408224.32 total 1102.40 last 80.23) - open/close 23087 (time 1786408234.32 total 1112.40 last 76.80) - open/close 23935 (time 1786408244.32 total 1122.40 last 84.80) - open/close 24651 (time 1786408254.32 total 1132.40 last 71.59) - open/close 25363 (time 1786408264.33 total 1142.41 last 71.14) - open/close 26018 (time 1786408274.33 total 1152.41 last 65.46) - open/close 26737 (time 1786408284.34 total 1162.42 last 71.88) - open/close 27461 (time 1786408294.34 total 1172.42 last 72.38) - open/close 28209 (time 1786408304.34 total 1182.42 last 74.76) - open/close 28842 (time 1786408314.37 total 1192.45 last 63.15) - open/close 29534 (time 1786408324.39 total 1202.47 last 69.08) - open/close 30000 (time 1786408330.63 total 1208.71 last 74.59) - open/close 30787 (time 1786408340.65 total 1218.73 last 78.60) - open/close 31572 (time 1786408350.65 total 1228.73 last 78.47) - open/close 32483 (time 1786408360.66 total 1238.74 last 91.01) - open/close 33315 (time 1786408370.67 total 1248.75 last 83.11) - open/close 34173 (time 1786408380.68 total 1258.76 last 85.74) - open/close 34928 (time 1786408390.69 total 1268.77 last 75.41) - open/close 35776 (time 1786408400.71 total 1278.79 last 84.64) - open/close 36519 (time 1786408410.72 total 1288.80 last 74.22) - open/close 37299 (time 1786408420.72 total 1298.80 last 77.97) - open/close 38125 (time 1786408430.73 total 1308.81 last 82.52) - open/close 38856 (time 1786408440.73 total 1318.81 last 73.10) - open/close 39626 (time 1786408450.74 total 1328.82 last 76.96) - open/close 40000 (time 1786408456.17 total 1334.25 last 68.82) - open/close 40855 (time 1786408466.17 total 1344.25 last 85.47) - open/close 41672 (time 1786408476.18 total 1354.26 last 81.64) - open/close 42327 (time 1786408486.19 total 1364.27 last 65.46) - open/close 42999 (time 1786408496.19 total 1374.27 last 67.16) - open/close 43750 (time 1786408506.20 total 1384.28 last 75.07) - open/close 44452 (time 1786408516.20 total 1394.28 last 70.17) - open/close 45361 (time 1786408526.21 total 1404.29 last 90.82) - open/close 46080 (time 1786408536.22 total 1414.30 last 71.88) - open/close 46781 (time 1786408546.22 total 1424.30 last 70.08) - open/close 47489 (time 1786408556.23 total 1434.31 last 70.71) - open/close 48231 (time 1786408566.24 total 1444.32 last 74.15) - open/close 48933 (time 1786408576.24 total 1454.32 last 70.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1464.19 seconds: 33.90 ops/second - unlinked 0 (time 1786408592 ; total 0 ; last 0) - unlinked 10000 (time 1786408726 ; total 134 ; last 134) - unlinked 20000 (time 1786408855 ; total 263 ; last 129) - unlinked 30000 (time 1786408988 ; total 396 ; last 133) - unlinked 40000 (time 1786409122 ; total 530 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 659 seconds: 75.312592 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server unloading modules via unload_modules_local on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local oleg249-server: modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2