== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:52:03 (1763344323) start mds service on oleg450-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg450-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 156 (time 1763344394.53 total 10.36 last 15.05) - open/close 364 (time 1763344405.20 total 21.04 last 19.49) - open/close 554 (time 1763344415.64 total 31.47 last 18.21) - open/close 781 (time 1763344426.13 total 41.97 last 21.63) - open/close 977 (time 1763344436.48 total 52.32 last 18.94) - open/close 1224 (time 1763344446.51 total 62.35 last 24.62) - open/close 1441 (time 1763344456.87 total 72.70 last 20.96) - open/close 1648 (time 1763344467.03 total 82.86 last 20.37) - open/close 1851 (time 1763344477.04 total 92.87 last 20.28) - open/close 2038 (time 1763344487.30 total 103.14 last 18.22) - open/close 2243 (time 1763344497.94 total 113.77 last 19.28) - open/close 2459 (time 1763344507.94 total 123.78 last 21.59) - open/close 2626 (time 1763344518.86 total 134.69 last 15.29) - open/close 2839 (time 1763344528.88 total 144.72 last 21.26) - open/close 3031 (time 1763344539.20 total 155.04 last 18.60) - open/close 3198 (time 1763344549.22 total 165.05 last 16.67) - open/close 3381 (time 1763344559.30 total 175.14 last 18.14) - open/close 3561 (time 1763344569.48 total 185.32 last 17.69) - open/close 3797 (time 1763344579.48 total 195.32 last 23.59) - open/close 3960 (time 1763344590.15 total 205.99 last 15.27) - open/close 4089 (time 1763344600.32 total 216.16 last 12.69) - open/close 4255 (time 1763344610.34 total 226.17 last 16.58) - open/close 4472 (time 1763344620.34 total 236.18 last 21.69) - open/close 4720 (time 1763344630.98 total 246.82 last 23.31) - open/close 4960 (time 1763344640.99 total 256.83 last 23.98) - open/close 5162 (time 1763344651.08 total 266.91 last 20.02) - open/close 5369 (time 1763344661.97 total 277.80 last 19.01) - open/close 5579 (time 1763344672.79 total 288.63 last 19.40) - open/close 5847 (time 1763344682.79 total 298.63 last 26.80) - open/close 6019 (time 1763344692.79 total 308.63 last 17.19) - open/close 6197 (time 1763344702.81 total 318.64 last 17.78) - open/close 6406 (time 1763344712.82 total 328.65 last 20.88) - open/close 6631 (time 1763344722.88 total 338.71 last 22.37) - open/close 6841 (time 1763344732.88 total 348.71 last 21.00) - open/close 7112 (time 1763344743.93 total 359.77 last 24.52) - open/close 7339 (time 1763344753.99 total 369.83 last 22.56) - open/close 7525 (time 1763344764.23 total 380.07 last 18.16) - open/close 7635 (time 1763344774.24 total 390.08 last 10.99) - open/close 7853 (time 1763344784.25 total 400.09 last 21.79) - open/close 8059 (time 1763344794.89 total 410.73 last 19.36) - open/close 8293 (time 1763344805.25 total 421.09 last 22.58) - open/close 8537 (time 1763344816.17 total 432.00 last 22.36) - open/close 8792 (time 1763344826.19 total 442.03 last 25.44) - open/close 9044 (time 1763344836.20 total 452.03 last 25.18) - open/close 9355 (time 1763344846.21 total 462.04 last 31.07) - open/close 9541 (time 1763344857.03 total 472.87 last 17.18) - open/close 9777 (time 1763344868.08 total 483.91 last 21.37) - open/close 9959 (time 1763344881.11 total 496.94 last 13.97) - open/close 10000 (time 1763344883.95 total 499.79 last 14.40) - open/close 10251 (time 1763344895.00 total 510.83 last 22.73) - open/close 10479 (time 1763344905.00 total 520.84 last 22.80) - open/close 10691 (time 1763344915.97 total 531.81 last 19.32) - open/close 10882 (time 1763344926.35 total 542.19 last 18.40) - open/close 11062 (time 1763344936.62 total 552.46 last 17.53) - open/close 11277 (time 1763344946.63 total 562.46 last 21.49) - open/close 11437 (time 1763344956.69 total 572.53 last 15.90) - open/close 11650 (time 1763344966.85 total 582.69 last 20.96) - open/close 11871 (time 1763344977.86 total 593.69 last 20.08) - open/close 12082 (time 1763344988.89 total 604.73 last 19.12) - open/close 12317 (time 1763344998.90 total 614.74 last 23.49) - open/close 12521 (time 1763345009.76 total 625.59 last 18.79) - open/close 12689 (time 1763345022.48 total 638.32 last 13.20) - open/close 12879 (time 1763345032.48 total 648.32 last 19.00) - open/close 13109 (time 1763345042.49 total 658.33 last 22.98) - open/close 13321 (time 1763345052.50 total 668.34 last 21.17) - open/close 13536 (time 1763345062.53 total 678.37 last 21.43) - open/close 13745 (time 1763345073.48 total 689.32 last 19.09) - open/close 13980 (time 1763345083.49 total 699.33 last 23.47) - open/close 14193 (time 1763345094.36 total 710.19 last 19.61) - open/close 14403 (time 1763345104.36 total 720.20 last 20.99) - open/close 14589 (time 1763345114.57 total 730.41 last 18.21) - open/close 14801 (time 1763345124.59 total 740.42 last 21.17) - open/close 15043 (time 1763345134.60 total 750.43 last 24.18) - open/close 15272 (time 1763345144.92 total 760.75 last 22.19) - open/close 15484 (time 1763345154.92 total 770.76 last 21.18) - open/close 15640 (time 1763345164.93 total 780.77 last 15.59) - open/close 15807 (time 1763345175.83 total 791.67 last 15.32) - open/close 16063 (time 1763345185.83 total 801.67 last 25.60) - open/close 16827 (time 1763345195.84 total 811.68 last 76.32) - open/close 17552 (time 1763345205.85 total 821.69 last 72.45) - open/close 18298 (time 1763345215.86 total 831.69 last 74.54) - open/close 19120 (time 1763345225.87 total 841.70 last 82.13) - open/close 19886 (time 1763345235.88 total 851.71 last 76.51) - open/close 20634 (time 1763345245.89 total 861.73 last 74.72) - open/close 21360 (time 1763345255.90 total 871.74 last 72.52) - open/close 22109 (time 1763345265.90 total 881.74 last 74.86) - open/close 22925 (time 1763345275.91 total 891.75 last 81.55) - open/close 23746 (time 1763345285.92 total 901.75 last 82.06) - open/close 24526 (time 1763345295.94 total 911.77 last 77.84) - open/close 25280 (time 1763345305.94 total 921.78 last 75.34) - open/close 26106 (time 1763345315.95 total 931.78 last 82.58) - open/close 26912 (time 1763345325.95 total 941.79 last 80.59) - open/close 27703 (time 1763345335.96 total 951.79 last 79.03) - open/close 28475 (time 1763345345.96 total 961.80 last 77.16) - open/close 29258 (time 1763345355.96 total 971.80 last 78.29) - open/close 29900 (time 1763345365.97 total 981.81 last 64.14) - open/close 30601 (time 1763345375.98 total 991.82 last 70.06) - open/close 31374 (time 1763345385.99 total 1001.83 last 77.21) - open/close 32122 (time 1763345396.00 total 1011.84 last 74.71) - open/close 32855 (time 1763345406.00 total 1021.84 last 73.28) - open/close 33607 (time 1763345416.01 total 1031.84 last 75.18) - open/close 34371 (time 1763345426.02 total 1041.85 last 76.32) - open/close 35150 (time 1763345436.03 total 1051.86 last 77.84) - open/close 35968 (time 1763345446.03 total 1061.87 last 81.76) - open/close 36680 (time 1763345456.04 total 1071.88 last 71.12) - open/close 37341 (time 1763345466.05 total 1081.88 last 66.07) - open/close 38103 (time 1763345476.05 total 1091.89 last 76.14) - open/close 38869 (time 1763345486.06 total 1101.90 last 76.52) - open/close 39615 (time 1763345496.08 total 1111.92 last 74.49) - open/close 40000 (time 1763345500.77 total 1116.60 last 82.15) - open/close 40757 (time 1763345510.78 total 1126.61 last 75.61) - open/close 41456 (time 1763345520.78 total 1136.62 last 69.84) - open/close 42160 (time 1763345530.80 total 1146.63 last 70.32) - open/close 42917 (time 1763345540.80 total 1156.63 last 75.69) - open/close 43630 (time 1763345550.80 total 1166.64 last 71.25) - open/close 44437 (time 1763345560.81 total 1176.65 last 80.67) - open/close 45220 (time 1763345570.82 total 1186.66 last 78.21) - open/close 46009 (time 1763345580.83 total 1196.66 last 78.86) - open/close 46782 (time 1763345590.83 total 1206.67 last 77.28) - open/close 47493 (time 1763345600.83 total 1216.67 last 71.09) - open/close 48236 (time 1763345610.84 total 1226.67 last 74.25) - open/close 49003 (time 1763345620.85 total 1236.68 last 76.63) - open/close 49722 (time 1763345630.85 total 1246.68 last 71.90) total: 50000 open/close in 1250.22 seconds: 39.99 ops/second - unlinked 0 (time 1763345638 ; total 0 ; last 0) - unlinked 10000 (time 1763345774 ; total 136 ; last 136) - unlinked 20000 (time 1763345911 ; total 273 ; last 137) - unlinked 30000 (time 1763346042 ; total 404 ; last 131) - unlinked 40000 (time 1763346178 ; total 540 ; last 136) total: 50000 unlinks in 673 seconds: 74.294205 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg450-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2