== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:53:53 (1767318833) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 151 (time 1767318980.64 total 10.01 last 15.08) - open/close 365 (time 1767318990.64 total 20.02 last 21.39) - open/close 588 (time 1767319000.66 total 30.03 last 22.27) - open/close 783 (time 1767319011.88 total 41.26 last 17.38) - open/close 1004 (time 1767319021.92 total 51.29 last 22.02) - open/close 1163 (time 1767319031.92 total 61.30 last 15.90) - open/close 1354 (time 1767319041.92 total 71.30 last 19.09) - open/close 1459 (time 1767319051.93 total 81.30 last 10.50) - open/close 1620 (time 1767319062.26 total 91.64 last 15.57) - open/close 1875 (time 1767319072.42 total 101.80 last 25.11) - open/close 2135 (time 1767319082.45 total 111.82 last 25.93) - open/close 2291 (time 1767319092.72 total 122.10 last 15.18) - open/close 2474 (time 1767319103.54 total 132.91 last 16.93) - open/close 2672 (time 1767319113.66 total 143.04 last 19.55) - open/close 2908 (time 1767319123.67 total 153.04 last 23.60) - open/close 3133 (time 1767319133.96 total 163.34 last 21.85) - open/close 3397 (time 1767319143.97 total 173.35 last 26.37) - open/close 3576 (time 1767319153.98 total 183.36 last 17.88) - open/close 3749 (time 1767319164.89 total 194.27 last 15.86) - open/close 3879 (time 1767319178.25 total 207.62 last 9.73) - open/close 4134 (time 1767319188.25 total 217.62 last 25.50) - open/close 4300 (time 1767319198.91 total 228.29 last 15.56) - open/close 4527 (time 1767319209.01 total 238.39 last 22.48) - open/close 4729 (time 1767319219.03 total 248.40 last 20.17) - open/close 4908 (time 1767319229.68 total 259.05 last 16.81) - open/close 5105 (time 1767319240.05 total 269.42 last 19.00) - open/close 5238 (time 1767319250.06 total 279.43 last 13.29) - open/close 5473 (time 1767319260.52 total 289.90 last 22.45) - open/close 5585 (time 1767319271.08 total 300.46 last 10.61) - open/close 5844 (time 1767319281.99 total 311.37 last 23.74) - open/close 6087 (time 1767319291.99 total 321.37 last 24.29) - open/close 6348 (time 1767319302.60 total 331.97 last 24.62) - open/close 6582 (time 1767319313.13 total 342.51 last 22.21) - open/close 6789 (time 1767319323.37 total 352.75 last 20.22) - open/close 7071 (time 1767319334.25 total 363.62 last 25.93) - open/close 7334 (time 1767319345.25 total 374.62 last 23.91) - open/close 7556 (time 1767319355.87 total 385.25 last 20.89) - open/close 7767 (time 1767319366.53 total 395.91 last 19.79) - open/close 8012 (time 1767319376.54 total 405.92 last 24.48) - open/close 8118 (time 1767319389.15 total 418.53 last 8.41) - open/close 8245 (time 1767319399.16 total 428.54 last 12.69) - open/close 8412 (time 1767319409.16 total 438.54 last 16.70) - open/close 8499 (time 1767319419.63 total 449.01 last 8.31) - open/close 8704 (time 1767319429.65 total 459.03 last 20.46) - open/close 8815 (time 1767319441.98 total 471.35 last 9.01) - open/close 9015 (time 1767319451.99 total 481.37 last 19.97) - open/close 9270 (time 1767319462.00 total 491.38 last 25.48) - open/close 9391 (time 1767319472.77 total 502.15 last 11.23) - open/close 9560 (time 1767319483.62 total 513.00 last 15.58) - open/close 9715 (time 1767319493.73 total 523.11 last 15.32) - open/close 9864 (time 1767319504.04 total 533.42 last 14.46) - open/close 9976 (time 1767319514.62 total 544.00 last 10.59) - open/close 10129 (time 1767319524.97 total 554.35 last 14.78) - open/close 10289 (time 1767319535.44 total 564.82 last 15.28) - open/close 10549 (time 1767319545.58 total 574.95 last 25.65) - open/close 10630 (time 1767319561.00 total 590.38 last 5.25) - open/close 10746 (time 1767319571.15 total 600.53 last 11.43) - open/close 10950 (time 1767319581.16 total 610.54 last 20.37) - open/close 11159 (time 1767319591.69 total 621.07 last 19.86) - open/close 11347 (time 1767319602.68 total 632.05 last 17.11) - open/close 11629 (time 1767319612.68 total 642.06 last 28.19) - open/close 11872 (time 1767319623.08 total 652.45 last 23.37) - open/close 11988 (time 1767319633.09 total 662.47 last 11.58) - open/close 12109 (time 1767319643.95 total 673.33 last 11.15) - open/close 12249 (time 1767319653.97 total 683.35 last 13.97) - open/close 12487 (time 1767319664.56 total 693.93 last 22.48) - open/close 12683 (time 1767319675.38 total 704.75 last 18.11) - open/close 12809 (time 1767319686.26 total 715.64 last 11.58) - open/close 12968 (time 1767319696.88 total 726.26 last 14.97) - open/close 13120 (time 1767319707.56 total 736.94 last 14.23) - open/close 13395 (time 1767319718.40 total 747.78 last 25.37) - open/close 13630 (time 1767319728.41 total 757.79 last 23.48) - open/close 13895 (time 1767319738.53 total 767.91 last 26.19) - open/close 14207 (time 1767319751.15 total 780.53 last 24.72) - open/close 14419 (time 1767319761.78 total 791.15 last 19.95) - open/close 14674 (time 1767319772.84 total 802.22 last 23.04) - open/close 14866 (time 1767319785.66 total 815.04 last 14.98) - open/close 15148 (time 1767319796.66 total 826.04 last 25.63) - open/close 15315 (time 1767319807.08 total 836.45 last 16.04) - open/close 15391 (time 1767319817.09 total 846.47 last 7.59) - open/close 15651 (time 1767319827.11 total 856.49 last 25.95) - open/close 15846 (time 1767319837.12 total 866.50 last 19.48) - open/close 16098 (time 1767319848.11 total 877.49 last 22.93) - open/close 16325 (time 1767319858.12 total 887.50 last 22.67) - open/close 16553 (time 1767319868.53 total 897.91 last 21.91) - open/close 16768 (time 1767319879.91 total 909.29 last 18.89) - open/close 17034 (time 1767319890.74 total 920.12 last 24.57) - open/close 17226 (time 1767319901.04 total 930.42 last 18.63) - open/close 17495 (time 1767319911.05 total 940.43 last 26.88) - open/close 17733 (time 1767319921.27 total 950.65 last 23.29) - open/close 17970 (time 1767319931.88 total 961.26 last 22.33) - open/close 18206 (time 1767319941.89 total 971.27 last 23.59) - open/close 18287 (time 1767319951.92 total 981.30 last 8.07) - open/close 18405 (time 1767319961.93 total 991.30 last 11.80) - open/close 18871 (time 1767319971.93 total 1001.31 last 46.58) - open/close 19693 (time 1767319981.93 total 1011.31 last 82.18) - open/close 20000 (time 1767319985.95 total 1015.33 last 76.37) - open/close 20832 (time 1767319995.95 total 1025.33 last 83.18) - open/close 21652 (time 1767320005.96 total 1035.34 last 81.93) - open/close 22404 (time 1767320015.97 total 1045.35 last 75.17) - open/close 23212 (time 1767320025.97 total 1055.35 last 80.79) - open/close 23896 (time 1767320035.97 total 1065.35 last 68.39) - open/close 24678 (time 1767320045.98 total 1075.35 last 78.15) - open/close 25449 (time 1767320055.98 total 1085.36 last 77.05) - open/close 26239 (time 1767320065.99 total 1095.37 last 78.92) - open/close 27182 (time 1767320075.99 total 1105.37 last 94.29) - open/close 27928 (time 1767320086.00 total 1115.38 last 74.55) - open/close 28763 (time 1767320096.06 total 1125.44 last 82.99) - open/close 29559 (time 1767320106.07 total 1135.45 last 79.52) - open/close 30000 (time 1767320111.79 total 1141.16 last 77.18) - open/close 30819 (time 1767320121.79 total 1151.16 last 81.89) - open/close 31580 (time 1767320131.79 total 1161.17 last 76.09) - open/close 32373 (time 1767320141.79 total 1171.17 last 79.27) - open/close 33235 (time 1767320151.80 total 1181.18 last 86.16) - open/close 33928 (time 1767320161.80 total 1191.18 last 69.30) - open/close 34756 (time 1767320171.81 total 1201.18 last 82.73) - open/close 35637 (time 1767320181.81 total 1211.19 last 88.06) - open/close 36420 (time 1767320191.81 total 1221.19 last 78.30) - open/close 37207 (time 1767320201.81 total 1231.19 last 78.69) - open/close 38005 (time 1767320211.82 total 1241.19 last 79.79) - open/close 38825 (time 1767320221.82 total 1251.19 last 81.99) - open/close 39602 (time 1767320231.83 total 1261.20 last 77.63) - open/close 40000 (time 1767320237.14 total 1266.52 last 74.85) - open/close 40755 (time 1767320247.15 total 1276.53 last 75.43) - open/close 41522 (time 1767320257.15 total 1286.53 last 76.69) - open/close 42457 (time 1767320267.16 total 1296.54 last 93.44) - open/close 43295 (time 1767320277.16 total 1306.54 last 83.79) - open/close 44060 (time 1767320287.17 total 1316.54 last 76.46) - open/close 44842 (time 1767320297.18 total 1326.56 last 78.06) - open/close 45676 (time 1767320307.19 total 1336.57 last 83.32) - open/close 46593 (time 1767320317.20 total 1346.58 last 91.64) - open/close 47298 (time 1767320327.21 total 1356.58 last 70.46) - open/close 48021 (time 1767320337.21 total 1366.59 last 72.26) - open/close 48765 (time 1767320347.22 total 1376.59 last 74.38) - open/close 49538 (time 1767320357.24 total 1386.62 last 77.10) total: 50000 open/close in 1391.29 seconds: 35.94 ops/second - unlinked 0 (time 1767320366 ; total 0 ; last 0) - unlinked 10000 (time 1767320495 ; total 129 ; last 129) - unlinked 20000 (time 1767320624 ; total 258 ; last 129) - unlinked 30000 (time 1767320748 ; total 382 ; last 124) - unlinked 40000 (time 1767320867 ; total 501 ; last 119) total: 50000 unlinks in 618 seconds: 80.906151 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg321-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2