== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:45:03 (1786171503) start mds service on oleg140-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg140-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 77 (time 1786171638.58 total 10.39 last 7.41) - open/close 198 (time 1786171648.81 total 20.62 last 11.83) - open/close 296 (time 1786171661.74 total 33.54 last 7.58) - open/close 385 (time 1786171672.14 total 43.95 last 8.56) - open/close 506 (time 1786171682.85 total 54.66 last 11.30) - open/close 695 (time 1786171692.85 total 64.66 last 18.89) - open/close 804 (time 1786171703.27 total 75.08 last 10.47) - open/close 944 (time 1786171713.30 total 85.11 last 13.96) - open/close 1087 (time 1786171724.29 total 96.10 last 13.01) - open/close 1276 (time 1786171734.31 total 106.12 last 18.87) - open/close 1374 (time 1786171745.15 total 116.96 last 9.04) - open/close 1568 (time 1786171756.11 total 127.92 last 17.71) - open/close 1657 (time 1786171767.03 total 138.84 last 8.15) - open/close 1775 (time 1786171779.51 total 151.32 last 9.46) - open/close 1914 (time 1786171789.52 total 161.33 last 13.89) - open/close 2024 (time 1786171800.58 total 172.39 last 9.95) - open/close 2205 (time 1786171811.26 total 183.07 last 16.94) - open/close 2306 (time 1786171821.28 total 193.09 last 10.09) - open/close 2433 (time 1786171831.29 total 203.10 last 12.68) - open/close 2559 (time 1786171842.31 total 214.12 last 11.44) - open/close 2686 (time 1786171852.56 total 224.36 last 12.39) - open/close 2836 (time 1786171862.56 total 234.37 last 14.99) - open/close 2956 (time 1786171872.57 total 244.38 last 11.98) - open/close 3069 (time 1786171883.00 total 254.80 last 10.84) - open/close 3275 (time 1786171895.37 total 267.18 last 16.65) - open/close 3482 (time 1786171906.06 total 277.87 last 19.36) - open/close 3770 (time 1786171916.42 total 288.23 last 27.80) - open/close 4007 (time 1786171927.01 total 298.82 last 22.39) - open/close 4206 (time 1786171937.17 total 308.98 last 19.58) - open/close 4393 (time 1786171947.28 total 319.09 last 18.51) - open/close 4687 (time 1786171957.28 total 329.09 last 29.39) - open/close 4816 (time 1786171967.61 total 339.42 last 12.49) - open/close 5053 (time 1786171977.61 total 349.42 last 23.70) - open/close 5318 (time 1786171987.61 total 359.42 last 26.50) - open/close 5495 (time 1786171998.21 total 370.01 last 16.70) - open/close 5566 (time 1786172009.01 total 380.81 last 6.57) - open/close 5812 (time 1786172019.84 total 391.65 last 22.70) - open/close 6065 (time 1786172029.85 total 401.66 last 25.29) - open/close 6254 (time 1786172040.07 total 411.88 last 18.49) - open/close 6500 (time 1786172050.30 total 422.11 last 24.05) - open/close 6710 (time 1786172060.54 total 432.35 last 20.50) - open/close 6934 (time 1786172071.42 total 443.23 last 20.59) - open/close 7036 (time 1786172083.73 total 455.54 last 8.28) - open/close 7292 (time 1786172094.55 total 466.36 last 23.67) - open/close 7400 (time 1786172104.90 total 476.71 last 10.44) - open/close 7678 (time 1786172114.94 total 486.74 last 27.70) - open/close 7864 (time 1786172126.78 total 498.59 last 15.70) - open/close 8001 (time 1786172139.09 total 510.89 last 11.14) - open/close 8208 (time 1786172149.25 total 521.05 last 20.37) - open/close 8449 (time 1786172159.93 total 531.74 last 22.56) - open/close 8662 (time 1786172170.09 total 541.90 last 20.97) - open/close 8883 (time 1786172180.11 total 551.91 last 22.06) - open/close 9051 (time 1786172190.11 total 561.92 last 16.80) - open/close 9186 (time 1786172200.25 total 572.06 last 13.31) - open/close 9329 (time 1786172210.31 total 582.12 last 14.21) - open/close 9532 (time 1786172221.51 total 593.32 last 18.12) - open/close 9676 (time 1786172231.52 total 603.33 last 14.39) - open/close 9795 (time 1786172241.53 total 613.34 last 11.88) - open/close 10000 (time 1786172249.14 total 620.95 last 26.95) - open/close 10161 (time 1786172259.65 total 631.46 last 15.32) - open/close 10353 (time 1786172269.84 total 641.65 last 18.84) - open/close 10527 (time 1786172280.92 total 652.73 last 15.70) - open/close 10797 (time 1786172291.91 total 663.72 last 24.57) - open/close 10921 (time 1786172302.92 total 674.72 last 11.27) - open/close 11026 (time 1786172315.65 total 687.46 last 8.25) - open/close 11190 (time 1786172325.84 total 697.65 last 16.08) - open/close 11414 (time 1786172335.86 total 707.67 last 22.36) - open/close 11680 (time 1786172345.87 total 717.68 last 26.59) - open/close 11946 (time 1786172356.76 total 728.57 last 24.43) - open/close 12095 (time 1786172367.68 total 739.49 last 13.64) - open/close 12247 (time 1786172377.95 total 749.76 last 14.80) - open/close 12289 (time 1786172388.34 total 760.14 last 4.04) - open/close 12556 (time 1786172399.47 total 771.28 last 23.98) - open/close 12818 (time 1786172410.43 total 782.24 last 23.90) - open/close 13081 (time 1786172421.45 total 793.26 last 23.88) - open/close 13296 (time 1786172432.26 total 804.07 last 19.89) - open/close 13480 (time 1786172442.82 total 814.63 last 17.42) - open/close 13680 (time 1786172453.70 total 825.51 last 18.38) - open/close 13863 (time 1786172463.73 total 835.54 last 18.25) - open/close 13975 (time 1786172474.58 total 846.38 last 10.33) - open/close 14142 (time 1786172484.58 total 856.39 last 16.70) - open/close 14256 (time 1786172494.71 total 866.52 last 11.25) - open/close 14391 (time 1786172505.64 total 877.44 last 12.36) - open/close 14486 (time 1786172515.66 total 887.46 last 9.48) - open/close 14632 (time 1786172525.68 total 897.49 last 14.57) - open/close 14897 (time 1786172536.25 total 908.06 last 25.07) - open/close 15140 (time 1786172546.57 total 918.38 last 23.55) - open/close 15310 (time 1786172559.62 total 931.43 last 13.02) - open/close 15508 (time 1786172569.62 total 941.43 last 19.80) - open/close 15719 (time 1786172579.64 total 951.45 last 21.07) - open/close 15898 (time 1786172592.19 total 963.99 last 14.27) - open/close 16095 (time 1786172602.37 total 974.18 last 19.35) - open/close 16332 (time 1786172612.37 total 984.18 last 23.70) - open/close 16441 (time 1786172627.03 total 998.84 last 7.44) - open/close 16518 (time 1786172637.18 total 1008.99 last 7.58) - open/close 16670 (time 1786172647.32 total 1019.13 last 14.99) - open/close 16809 (time 1786172659.42 total 1031.23 last 11.49) - open/close 16942 (time 1786172669.44 total 1041.25 last 13.27) - open/close 17011 (time 1786172680.19 total 1052.00 last 6.42) - open/close 17123 (time 1786172690.95 total 1062.76 last 10.41) - open/close 17741 (time 1786172700.96 total 1072.77 last 61.78) - open/close 18514 (time 1786172710.96 total 1082.77 last 77.26) - open/close 19210 (time 1786172720.96 total 1092.77 last 69.60) - open/close 19942 (time 1786172730.97 total 1102.78 last 73.14) - open/close 20514 (time 1786172740.98 total 1112.79 last 57.17) - open/close 21244 (time 1786172750.98 total 1122.79 last 72.94) - open/close 21962 (time 1786172760.99 total 1132.80 last 71.76) - open/close 22781 (time 1786172771.00 total 1142.81 last 81.83) - open/close 23461 (time 1786172781.00 total 1152.81 last 67.97) - open/close 24128 (time 1786172791.01 total 1162.81 last 66.68) - open/close 24808 (time 1786172801.01 total 1172.82 last 67.96) - open/close 25497 (time 1786172811.02 total 1182.83 last 68.87) - open/close 26325 (time 1786172821.03 total 1192.84 last 82.71) - open/close 27129 (time 1786172831.03 total 1202.84 last 80.37) - open/close 27768 (time 1786172841.04 total 1212.85 last 63.81) - open/close 28421 (time 1786172851.05 total 1222.86 last 65.26) - open/close 29108 (time 1786172861.06 total 1232.87 last 68.66) - open/close 29791 (time 1786172871.07 total 1242.87 last 68.25) - open/close 30000 (time 1786172874.54 total 1246.35 last 60.19) - open/close 30697 (time 1786172884.54 total 1256.35 last 69.67) - open/close 31420 (time 1786172894.56 total 1266.37 last 72.19) - open/close 32154 (time 1786172904.56 total 1276.37 last 73.39) - open/close 32930 (time 1786172914.56 total 1286.37 last 77.59) - open/close 33683 (time 1786172924.58 total 1296.39 last 75.15) - open/close 34473 (time 1786172934.60 total 1306.40 last 78.87) - open/close 35217 (time 1786172944.60 total 1316.41 last 74.37) - open/close 36042 (time 1786172954.61 total 1326.42 last 82.44) - open/close 36845 (time 1786172964.63 total 1336.44 last 80.15) - open/close 37576 (time 1786172974.65 total 1346.46 last 72.92) - open/close 38376 (time 1786172984.66 total 1356.47 last 79.96) - open/close 39371 (time 1786172994.67 total 1366.48 last 99.40) - open/close 40000 (time 1786173001.41 total 1373.22 last 93.24) - open/close 40800 (time 1786173011.43 total 1383.24 last 79.84) - open/close 41461 (time 1786173021.44 total 1393.25 last 66.06) - open/close 42190 (time 1786173031.44 total 1403.25 last 72.87) - open/close 42955 (time 1786173041.45 total 1413.26 last 76.45) - open/close 43787 (time 1786173051.46 total 1423.26 last 83.16) - open/close 44573 (time 1786173061.46 total 1433.27 last 78.59) - open/close 45386 (time 1786173071.46 total 1443.27 last 81.24) - open/close 46233 (time 1786173081.47 total 1453.27 last 84.69) - open/close 47008 (time 1786173091.48 total 1463.28 last 77.42) - open/close 47756 (time 1786173101.48 total 1473.29 last 74.74) - open/close 48579 (time 1786173111.49 total 1483.30 last 82.23) - open/close 49281 (time 1786173121.50 total 1493.30 last 70.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1499.12 seconds: 33.11 ops/second - unlinked 0 (time 1786173132 ; total 0 ; last 0) - unlinked 10000 (time 1786173258 ; total 126 ; last 126) - unlinked 20000 (time 1786173406 ; total 274 ; last 148) - unlinked 30000 (time 1786173542 ; total 410 ; last 136) - unlinked 40000 (time 1786173679 ; total 547 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 678 seconds: 73.202065 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: modules unloaded. pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2