== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:03:01 (1763341381) start mds service on oleg403-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1763341454.06 total 10.64 last 20.12) - open/close 345 (time 1763341464.08 total 20.65 last 13.08) - open/close 575 (time 1763341474.08 total 30.65 last 23.00) - open/close 748 (time 1763341486.84 total 43.42 last 13.55) - open/close 986 (time 1763341497.05 total 53.62 last 23.33) - open/close 1161 (time 1763341507.07 total 63.64 last 17.47) - open/close 1350 (time 1763341518.05 total 74.62 last 17.21) - open/close 1532 (time 1763341528.58 total 85.15 last 17.28) - open/close 1655 (time 1763341538.59 total 95.16 last 12.29) - open/close 1800 (time 1763341549.09 total 105.66 last 13.81) - open/close 2076 (time 1763341560.17 total 116.74 last 24.91) - open/close 2218 (time 1763341570.29 total 126.87 last 14.03) - open/close 2452 (time 1763341581.03 total 137.60 last 21.80) - open/close 2665 (time 1763341591.97 total 148.54 last 19.46) - open/close 2895 (time 1763341601.98 total 158.55 last 22.99) - open/close 3098 (time 1763341612.33 total 168.90 last 19.62) - open/close 3342 (time 1763341622.34 total 178.92 last 24.36) - open/close 3516 (time 1763341633.40 total 189.98 last 15.73) - open/close 3648 (time 1763341643.41 total 199.98 last 13.20) - open/close 3864 (time 1763341653.41 total 209.98 last 21.59) - open/close 4054 (time 1763341664.43 total 221.00 last 17.25) - open/close 4282 (time 1763341675.30 total 231.87 last 20.97) - open/close 4528 (time 1763341685.49 total 242.07 last 24.13) - open/close 4672 (time 1763341696.04 total 252.61 last 13.66) - open/close 4923 (time 1763341706.50 total 263.08 last 23.99) - open/close 5080 (time 1763341716.51 total 273.09 last 15.68) - open/close 5233 (time 1763341726.58 total 283.15 last 15.20) - open/close 5423 (time 1763341736.59 total 293.16 last 18.99) - open/close 5625 (time 1763341747.12 total 303.69 last 19.18) - open/close 5840 (time 1763341757.14 total 313.71 last 21.46) - open/close 6024 (time 1763341768.23 total 324.80 last 16.59) - open/close 6130 (time 1763341778.26 total 334.84 last 10.56) - open/close 6316 (time 1763341789.72 total 346.30 last 16.23) - open/close 6521 (time 1763341799.77 total 356.35 last 20.39) - open/close 6744 (time 1763341810.66 total 367.23 last 20.48) - open/close 6920 (time 1763341820.68 total 377.25 last 17.57) - open/close 7085 (time 1763341834.47 total 391.05 last 11.96) - open/close 7312 (time 1763341845.22 total 401.79 last 21.13) - open/close 7556 (time 1763341855.23 total 411.80 last 24.37) - open/close 7682 (time 1763341865.83 total 422.40 last 11.89) - open/close 7815 (time 1763341880.06 total 436.63 last 9.35) - open/close 8040 (time 1763341890.06 total 446.64 last 22.49) - open/close 8188 (time 1763341900.96 total 457.53 last 13.58) - open/close 8353 (time 1763341911.59 total 468.16 last 15.52) - open/close 8493 (time 1763341921.97 total 478.54 last 13.49) - open/close 8653 (time 1763341931.99 total 488.56 last 15.97) - open/close 8833 (time 1763341942.02 total 498.59 last 17.94) - open/close 9064 (time 1763341952.05 total 508.62 last 23.03) - open/close 9317 (time 1763341962.08 total 518.65 last 25.23) - open/close 9518 (time 1763341972.20 total 528.77 last 19.86) - open/close 9718 (time 1763341982.24 total 538.81 last 19.92) - open/close 9962 (time 1763341992.61 total 549.18 last 23.53) - open/close 10185 (time 1763342003.12 total 559.69 last 21.21) - open/close 10421 (time 1763342015.59 total 572.17 last 18.92) - open/close 10621 (time 1763342025.78 total 582.35 last 19.64) - open/close 10869 (time 1763342036.27 total 592.85 last 23.63) - open/close 11103 (time 1763342046.28 total 602.86 last 23.38) - open/close 11275 (time 1763342057.13 total 613.70 last 15.86) - open/close 11484 (time 1763342067.74 total 624.31 last 19.69) - open/close 11703 (time 1763342078.36 total 634.94 last 20.62) - open/close 11951 (time 1763342089.19 total 645.76 last 22.91) - open/close 12165 (time 1763342099.19 total 655.77 last 21.39) - open/close 12381 (time 1763342109.20 total 665.77 last 21.58) - open/close 12563 (time 1763342119.21 total 675.78 last 18.18) - open/close 12808 (time 1763342130.28 total 686.85 last 22.13) - open/close 12994 (time 1763342141.29 total 697.87 last 16.89) - open/close 13193 (time 1763342152.23 total 708.80 last 18.20) - open/close 13400 (time 1763342162.27 total 718.85 last 20.61) - open/close 13604 (time 1763342172.27 total 728.85 last 20.40) - open/close 13830 (time 1763342182.69 total 739.26 last 21.71) - open/close 14082 (time 1763342192.69 total 749.26 last 25.19) - open/close 14273 (time 1763342204.83 total 761.40 last 15.73) - open/close 14484 (time 1763342214.83 total 771.40 last 21.09) - open/close 14692 (time 1763342224.84 total 781.42 last 20.78) - open/close 14882 (time 1763342234.86 total 791.43 last 18.97) - open/close 15084 (time 1763342245.02 total 801.59 last 19.88) - open/close 15297 (time 1763342255.80 total 812.37 last 19.75) - open/close 15521 (time 1763342265.80 total 822.38 last 22.39) - open/close 15722 (time 1763342276.06 total 832.63 last 19.60) - open/close 15934 (time 1763342286.82 total 843.39 last 19.70) - open/close 16200 (time 1763342296.94 total 853.52 last 26.27) - open/close 16425 (time 1763342306.95 total 863.52 last 22.49) - open/close 16635 (time 1763342317.33 total 873.90 last 20.24) - open/close 16858 (time 1763342327.91 total 884.48 last 21.08) - open/close 17069 (time 1763342338.40 total 894.97 last 20.10) - open/close 17315 (time 1763342348.41 total 904.98 last 24.58) - open/close 17545 (time 1763342359.46 total 916.03 last 20.82) - open/close 17749 (time 1763342369.73 total 926.30 last 19.87) - open/close 18135 (time 1763342379.75 total 936.32 last 38.52) - open/close 18702 (time 1763342389.76 total 946.33 last 56.65) - open/close 19376 (time 1763342399.77 total 956.34 last 67.33) - open/close 19994 (time 1763342409.77 total 966.34 last 61.78) - open/close 20558 (time 1763342419.78 total 976.35 last 56.37) - open/close 21267 (time 1763342429.79 total 986.37 last 70.79) - open/close 21893 (time 1763342439.80 total 996.37 last 62.56) - open/close 22574 (time 1763342449.81 total 1006.38 last 68.04) - open/close 23270 (time 1763342459.82 total 1016.39 last 69.52) - open/close 23990 (time 1763342469.84 total 1026.41 last 71.88) - open/close 24733 (time 1763342479.84 total 1036.42 last 74.25) - open/close 25454 (time 1763342489.85 total 1046.43 last 72.03) - open/close 26176 (time 1763342499.86 total 1056.43 last 72.15) - open/close 26845 (time 1763342509.86 total 1066.44 last 66.89) - open/close 27531 (time 1763342519.88 total 1076.45 last 68.50) - open/close 28099 (time 1763342529.90 total 1086.47 last 56.69) - open/close 28707 (time 1763342539.90 total 1096.47 last 60.78) - open/close 29338 (time 1763342549.90 total 1106.48 last 63.07) - open/close 29994 (time 1763342559.91 total 1116.48 last 65.59) - open/close 30647 (time 1763342569.91 total 1126.48 last 65.27) - open/close 31314 (time 1763342579.92 total 1136.50 last 66.62) - open/close 31912 (time 1763342589.93 total 1146.50 last 59.77) - open/close 32489 (time 1763342599.93 total 1156.50 last 57.69) - open/close 33094 (time 1763342609.93 total 1166.50 last 60.49) - open/close 33784 (time 1763342619.93 total 1176.50 last 68.99) - open/close 34465 (time 1763342629.95 total 1186.52 last 68.00) - open/close 35141 (time 1763342639.96 total 1196.54 last 67.49) - open/close 35849 (time 1763342649.99 total 1206.56 last 70.63) - open/close 36510 (time 1763342660.00 total 1216.57 last 66.04) - open/close 37162 (time 1763342670.00 total 1226.57 last 65.18) - open/close 37812 (time 1763342680.01 total 1236.58 last 64.93) - open/close 38502 (time 1763342690.02 total 1246.59 last 68.96) - open/close 39127 (time 1763342700.02 total 1256.60 last 62.46) - open/close 39763 (time 1763342710.02 total 1266.60 last 63.60) - open/close 40000 (time 1763342713.56 total 1270.13 last 67.06) - open/close 40737 (time 1763342723.56 total 1280.14 last 73.65) - open/close 41424 (time 1763342733.57 total 1290.15 last 68.65) - open/close 42108 (time 1763342743.57 total 1300.15 last 68.38) - open/close 42811 (time 1763342753.58 total 1310.16 last 70.24) - open/close 43468 (time 1763342763.59 total 1320.16 last 65.66) - open/close 44122 (time 1763342773.59 total 1330.16 last 65.40) - open/close 44729 (time 1763342783.60 total 1340.18 last 60.62) - open/close 45419 (time 1763342793.61 total 1350.18 last 68.96) - open/close 46127 (time 1763342803.62 total 1360.19 last 70.74) - open/close 46831 (time 1763342813.63 total 1370.20 last 70.30) - open/close 47536 (time 1763342823.63 total 1380.20 last 70.49) - open/close 48178 (time 1763342833.63 total 1390.21 last 64.20) - open/close 48871 (time 1763342843.64 total 1400.21 last 69.25) - open/close 49533 (time 1763342853.64 total 1410.22 last 66.16) total: 50000 open/close in 1417.13 seconds: 35.28 ops/second - unlinked 0 (time 1763342864 ; total 0 ; last 0) - unlinked 10000 (time 1763343012 ; total 148 ; last 148) - unlinked 20000 (time 1763343159 ; total 295 ; last 147) - unlinked 30000 (time 1763343307 ; total 443 ; last 148) - unlinked 40000 (time 1763343462 ; total 598 ; last 155) total: 50000 unlinks in 745 seconds: 67.114098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2