== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:21:26 (1753744886) start mds service on oleg237-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1753744955.87 total 11.00 last 19.46) - open/close 391 (time 1753744965.87 total 21.00 last 17.69) - open/close 595 (time 1753744976.49 total 31.62 last 19.21) - open/close 790 (time 1753744986.92 total 42.04 last 18.71) - open/close 993 (time 1753744996.92 total 52.05 last 20.29) - open/close 1196 (time 1753745006.93 total 62.06 last 20.28) - open/close 1410 (time 1753745017.25 total 72.38 last 20.73) - open/close 1609 (time 1753745027.69 total 82.82 last 19.06) - open/close 1792 (time 1753745038.51 total 93.64 last 16.92) - open/close 2014 (time 1753745048.90 total 104.02 last 21.37) - open/close 2207 (time 1753745059.61 total 114.74 last 18.01) - open/close 2418 (time 1753745070.37 total 125.49 last 19.62) - open/close 2634 (time 1753745080.39 total 135.52 last 21.54) - open/close 2841 (time 1753745090.41 total 145.54 last 20.67) - open/close 3069 (time 1753745100.53 total 155.66 last 22.53) - open/close 3292 (time 1753745110.54 total 165.67 last 22.27) - open/close 3496 (time 1753745120.58 total 175.71 last 20.32) - open/close 3696 (time 1753745130.59 total 185.72 last 19.97) - open/close 3897 (time 1753745140.61 total 195.74 last 20.07) - open/close 4115 (time 1753745150.88 total 206.01 last 21.22) - open/close 4322 (time 1753745160.88 total 216.01 last 20.70) - open/close 4492 (time 1753745171.38 total 226.51 last 16.19) - open/close 4681 (time 1753745182.12 total 237.25 last 17.61) - open/close 4901 (time 1753745192.13 total 247.26 last 21.98) - open/close 5107 (time 1753745202.13 total 257.26 last 20.59) - open/close 5344 (time 1753745212.21 total 267.34 last 23.51) - open/close 5561 (time 1753745222.22 total 277.34 last 21.70) - open/close 5734 (time 1753745232.24 total 287.37 last 17.26) - open/close 5910 (time 1753745242.56 total 297.69 last 17.05) - open/close 6145 (time 1753745253.41 total 308.54 last 21.66) - open/close 6326 (time 1753745263.42 total 318.55 last 18.09) - open/close 6497 (time 1753745276.32 total 331.45 last 13.25) - open/close 6631 (time 1753745287.47 total 342.60 last 12.03) - open/close 6868 (time 1753745298.37 total 353.50 last 21.74) - open/close 7093 (time 1753745309.23 total 364.36 last 20.71) - open/close 7301 (time 1753745319.46 total 374.59 last 20.34) - open/close 7515 (time 1753745329.48 total 384.60 last 21.37) - open/close 7732 (time 1753745339.48 total 394.60 last 21.70) - open/close 7907 (time 1753745349.86 total 404.99 last 16.85) - open/close 8114 (time 1753745359.88 total 415.00 last 20.68) - open/close 8379 (time 1753745370.48 total 425.61 last 24.99) - open/close 8622 (time 1753745380.83 total 435.96 last 23.46) - open/close 8864 (time 1753745390.83 total 445.96 last 24.20) - open/close 9056 (time 1753745400.84 total 455.97 last 19.18) - open/close 9289 (time 1753745411.86 total 466.99 last 21.15) - open/close 9519 (time 1753745421.86 total 476.99 last 22.99) - open/close 9745 (time 1753745432.67 total 487.80 last 20.91) - open/close 9946 (time 1753745442.79 total 497.92 last 19.87) - open/close 10000 (time 1753745444.87 total 500.00 last 25.91) - open/close 10191 (time 1753745455.09 total 510.21 last 18.70) - open/close 10391 (time 1753745465.58 total 520.71 last 19.05) - open/close 10639 (time 1753745475.59 total 530.72 last 24.79) - open/close 10837 (time 1753745485.99 total 541.12 last 19.04) - open/close 11019 (time 1753745496.63 total 551.76 last 17.10) - open/close 11199 (time 1753745506.64 total 561.77 last 17.98) - open/close 11437 (time 1753745517.38 total 572.50 last 22.17) - open/close 11649 (time 1753745527.47 total 582.60 last 21.00) - open/close 11846 (time 1753745537.47 total 592.60 last 19.69) - open/close 12063 (time 1753745547.48 total 602.61 last 21.69) - open/close 12279 (time 1753745557.87 total 613.00 last 20.79) - open/close 12490 (time 1753745567.89 total 623.02 last 21.06) - open/close 12695 (time 1753745578.53 total 633.66 last 19.27) - open/close 12915 (time 1753745588.74 total 643.87 last 21.54) - open/close 13114 (time 1753745598.76 total 653.89 last 19.86) - open/close 13344 (time 1753745609.58 total 664.70 last 21.27) - open/close 13590 (time 1753745620.55 total 675.68 last 22.42) - open/close 13813 (time 1753745631.04 total 686.16 last 21.26) - open/close 14022 (time 1753745641.41 total 696.54 last 20.14) - open/close 14231 (time 1753745651.94 total 707.07 last 19.84) - open/close 14426 (time 1753745662.19 total 717.32 last 19.03) - open/close 14641 (time 1753745672.35 total 727.48 last 21.15) - open/close 14865 (time 1753745682.92 total 738.05 last 21.20) - open/close 15067 (time 1753745692.93 total 748.06 last 20.18) - open/close 15255 (time 1753745703.53 total 758.66 last 17.75) - open/close 15380 (time 1753745713.91 total 769.04 last 12.04) - open/close 15539 (time 1753745724.84 total 779.97 last 14.55) - open/close 15715 (time 1753745734.97 total 790.09 last 17.38) - open/close 15917 (time 1753745744.97 total 800.09 last 20.20) - open/close 16116 (time 1753745754.97 total 810.09 last 19.90) - open/close 16283 (time 1753745764.98 total 820.10 last 16.68) - open/close 16443 (time 1753745775.78 total 830.91 last 14.81) - open/close 16927 (time 1753745785.79 total 840.92 last 48.34) - open/close 17694 (time 1753745795.80 total 850.93 last 76.68) - open/close 18392 (time 1753745805.81 total 860.94 last 69.71) - open/close 19064 (time 1753745815.82 total 870.95 last 67.15) - open/close 19742 (time 1753745825.82 total 880.95 last 67.75) - open/close 20000 (time 1753745829.55 total 884.67 last 69.35) - open/close 20675 (time 1753745839.55 total 894.68 last 67.44) - open/close 21326 (time 1753745849.57 total 904.70 last 64.98) - open/close 21957 (time 1753745859.58 total 914.71 last 63.02) - open/close 22696 (time 1753745869.59 total 924.71 last 73.89) - open/close 23289 (time 1753745879.60 total 934.73 last 59.22) - open/close 23898 (time 1753745889.61 total 944.73 last 60.86) - open/close 24520 (time 1753745899.61 total 954.74 last 62.18) - open/close 25214 (time 1753745909.61 total 964.74 last 69.38) - open/close 25874 (time 1753745919.61 total 974.74 last 65.99) - open/close 26526 (time 1753745929.62 total 984.75 last 65.16) - open/close 27224 (time 1753745939.62 total 994.75 last 69.78) - open/close 27959 (time 1753745949.63 total 1004.75 last 73.48) - open/close 28702 (time 1753745959.64 total 1014.76 last 74.23) - open/close 29559 (time 1753745969.64 total 1024.77 last 85.68) - open/close 30000 (time 1753745975.41 total 1030.54 last 76.36) - open/close 30858 (time 1753745985.42 total 1040.55 last 85.75) - open/close 31611 (time 1753745995.42 total 1050.55 last 75.27) - open/close 32355 (time 1753746005.43 total 1060.56 last 74.32) - open/close 33089 (time 1753746015.45 total 1070.57 last 73.31) - open/close 33838 (time 1753746025.45 total 1080.57 last 74.90) - open/close 34555 (time 1753746035.45 total 1090.58 last 71.64) - open/close 35313 (time 1753746045.45 total 1100.58 last 75.79) - open/close 36111 (time 1753746055.47 total 1110.60 last 79.66) - open/close 36904 (time 1753746065.49 total 1120.62 last 79.17) - open/close 37805 (time 1753746075.50 total 1130.63 last 90.01) - open/close 38643 (time 1753746085.50 total 1140.63 last 83.79) - open/close 39493 (time 1753746095.50 total 1150.63 last 84.97) - open/close 40000 (time 1753746101.71 total 1156.84 last 81.66) - open/close 40759 (time 1753746111.71 total 1166.84 last 75.89) - open/close 41496 (time 1753746121.72 total 1176.85 last 73.63) - open/close 42294 (time 1753746131.73 total 1186.86 last 79.77) - open/close 43082 (time 1753746141.74 total 1196.86 last 78.73) - open/close 43832 (time 1753746151.75 total 1206.88 last 74.92) - open/close 44671 (time 1753746161.76 total 1216.88 last 83.83) - open/close 45412 (time 1753746171.76 total 1226.89 last 74.06) - open/close 46131 (time 1753746181.77 total 1236.90 last 71.84) - open/close 46930 (time 1753746191.78 total 1246.91 last 79.84) - open/close 47845 (time 1753746201.78 total 1256.91 last 91.46) - open/close 48662 (time 1753746211.79 total 1266.91 last 81.67) - open/close 49461 (time 1753746221.79 total 1276.92 last 79.88) total: 50000 open/close in 1282.81 seconds: 38.98 ops/second - unlinked 0 (time 1753746230 ; total 0 ; last 0) - unlinked 10000 (time 1753746338 ; total 108 ; last 108) - unlinked 20000 (time 1753746443 ; total 213 ; last 105) - unlinked 30000 (time 1753746523 ; total 293 ; last 80) - unlinked 40000 (time 1753746583 ; total 353 ; last 60) total: 50000 unlinks in 413 seconds: 121.065376 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2