== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:08:51 (1763334531) start mds service on oleg249-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg249-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg249-server: detected 4 online CPUs by sysfs oleg249-server: MODOPTS_LIBCFS= oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg249-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg249-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1763334602.51 total 10.45 last 20.68) - open/close 426 (time 1763334613.08 total 21.02 last 19.86) - open/close 626 (time 1763334623.53 total 31.47 last 19.14) - open/close 859 (time 1763334633.54 total 41.48 last 23.28) - open/close 1041 (time 1763334643.67 total 51.61 last 17.96) - open/close 1216 (time 1763334653.68 total 61.62 last 17.48) - open/close 1443 (time 1763334663.95 total 71.89 last 22.11) - open/close 1663 (time 1763334673.95 total 81.89 last 21.99) - open/close 1923 (time 1763334684.27 total 92.21 last 25.20) - open/close 2064 (time 1763334694.79 total 102.73 last 13.40) - open/close 2228 (time 1763334705.55 total 113.49 last 15.25) - open/close 2390 (time 1763334716.12 total 124.06 last 15.32) - open/close 2548 (time 1763334726.41 total 134.35 last 15.35) - open/close 2747 (time 1763334736.60 total 144.53 last 19.54) - open/close 2909 (time 1763334746.89 total 154.83 last 15.73) - open/close 3105 (time 1763334756.89 total 164.83 last 19.60) - open/close 3249 (time 1763334767.19 total 175.13 last 13.99) - open/close 3495 (time 1763334777.82 total 185.76 last 23.14) - open/close 3727 (time 1763334788.43 total 196.37 last 21.88) - open/close 3859 (time 1763334801.42 total 209.36 last 10.16) - open/close 4067 (time 1763334811.43 total 219.37 last 20.79) - open/close 4201 (time 1763334821.90 total 229.84 last 12.80) - open/close 4485 (time 1763334832.60 total 240.54 last 26.53) - open/close 4667 (time 1763334842.61 total 250.54 last 18.20) - open/close 4801 (time 1763334852.66 total 260.59 last 13.33) - open/close 4967 (time 1763334863.38 total 271.32 last 15.47) - open/close 5144 (time 1763334873.48 total 281.42 last 17.52) - open/close 5348 (time 1763334883.72 total 291.66 last 19.93) - open/close 5522 (time 1763334893.74 total 301.67 last 17.37) - open/close 5667 (time 1763334904.21 total 312.15 last 13.85) - open/close 5839 (time 1763334914.22 total 322.15 last 17.19) - open/close 6035 (time 1763334925.08 total 333.01 last 18.05) - open/close 6178 (time 1763334935.52 total 343.46 last 13.69) - open/close 6336 (time 1763334946.00 total 353.94 last 15.08) - open/close 6483 (time 1763334956.88 total 364.82 last 13.51) - open/close 6688 (time 1763334966.88 total 374.82 last 20.49) - open/close 6931 (time 1763334977.36 total 385.30 last 23.18) - open/close 7154 (time 1763334987.39 total 395.33 last 22.25) - open/close 7359 (time 1763334997.64 total 405.58 last 19.99) - open/close 7501 (time 1763335007.84 total 415.77 last 13.93) - open/close 7681 (time 1763335018.72 total 426.65 last 16.54) - open/close 7865 (time 1763335028.73 total 436.67 last 18.38) - open/close 8085 (time 1763335038.74 total 446.68 last 21.98) - open/close 8313 (time 1763335049.55 total 457.49 last 21.09) - open/close 8547 (time 1763335059.56 total 467.50 last 23.37) - open/close 8826 (time 1763335069.57 total 477.51 last 27.88) - open/close 9007 (time 1763335079.69 total 487.63 last 17.89) - open/close 9200 (time 1763335090.72 total 498.66 last 17.50) - open/close 9407 (time 1763335100.74 total 508.67 last 20.66) - open/close 9635 (time 1763335110.87 total 518.81 last 22.49) - open/close 9913 (time 1763335121.73 total 529.67 last 25.61) - open/close 10000 (time 1763335125.35 total 533.29 last 24.02) - open/close 10184 (time 1763335135.36 total 543.30 last 18.38) - open/close 10367 (time 1763335146.39 total 554.33 last 16.60) - open/close 10582 (time 1763335157.01 total 564.95 last 20.24) - open/close 10699 (time 1763335167.06 total 574.99 last 11.65) - open/close 10850 (time 1763335177.07 total 585.01 last 15.08) - open/close 10999 (time 1763335189.27 total 597.21 last 12.21) - open/close 11187 (time 1763335199.31 total 607.25 last 18.73) - open/close 11480 (time 1763335210.33 total 618.27 last 26.59) - open/close 11752 (time 1763335221.14 total 629.08 last 25.16) - open/close 12008 (time 1763335231.16 total 639.10 last 25.55) - open/close 12139 (time 1763335242.20 total 650.13 last 11.87) - open/close 12277 (time 1763335252.43 total 660.37 last 13.48) - open/close 12471 (time 1763335262.44 total 670.38 last 19.38) - open/close 12664 (time 1763335272.46 total 680.40 last 19.27) - open/close 12851 (time 1763335283.09 total 691.03 last 17.60) - open/close 13516 (time 1763335293.09 total 701.03 last 66.47) - open/close 14350 (time 1763335303.10 total 711.04 last 83.35) - open/close 15175 (time 1763335313.10 total 721.04 last 82.47) - open/close 16133 (time 1763335323.12 total 731.06 last 95.64) - open/close 16960 (time 1763335333.13 total 741.07 last 82.59) - open/close 17813 (time 1763335343.13 total 751.07 last 85.29) - open/close 18639 (time 1763335353.14 total 761.08 last 82.52) - open/close 19455 (time 1763335363.14 total 771.08 last 81.59) - open/close 20000 (time 1763335369.40 total 777.34 last 87.09) - open/close 20904 (time 1763335379.41 total 787.35 last 90.33) - open/close 21786 (time 1763335389.42 total 797.36 last 88.12) - open/close 22545 (time 1763335399.42 total 807.36 last 75.85) - open/close 23381 (time 1763335409.44 total 817.37 last 83.51) - open/close 24276 (time 1763335419.44 total 827.38 last 89.47) - open/close 25137 (time 1763335429.44 total 837.38 last 86.05) - open/close 25928 (time 1763335439.45 total 847.38 last 79.08) - open/close 26735 (time 1763335449.45 total 857.39 last 80.69) - open/close 27525 (time 1763335459.45 total 867.39 last 78.95) - open/close 28287 (time 1763335469.45 total 877.39 last 76.19) - open/close 29057 (time 1763335479.47 total 887.40 last 76.92) - open/close 29895 (time 1763335489.47 total 897.40 last 83.80) - open/close 30704 (time 1763335499.48 total 907.42 last 80.76) - open/close 31502 (time 1763335509.49 total 917.42 last 79.77) - open/close 32306 (time 1763335519.49 total 927.43 last 80.37) - open/close 33186 (time 1763335529.50 total 937.44 last 87.90) - open/close 33986 (time 1763335539.51 total 947.45 last 79.94) - open/close 34728 (time 1763335549.51 total 957.45 last 74.20) - open/close 35497 (time 1763335559.51 total 967.45 last 76.88) - open/close 36271 (time 1763335569.53 total 977.47 last 77.28) - open/close 37074 (time 1763335579.53 total 987.47 last 80.28) - open/close 37851 (time 1763335589.53 total 997.47 last 77.69) - open/close 38710 (time 1763335599.54 total 1007.48 last 85.85) - open/close 39537 (time 1763335609.55 total 1017.49 last 82.60) - open/close 40000 (time 1763335614.89 total 1022.83 last 86.71) - open/close 40850 (time 1763335624.89 total 1032.83 last 84.97) - open/close 41680 (time 1763335634.90 total 1042.83 last 82.96) - open/close 42496 (time 1763335644.90 total 1052.84 last 81.55) - open/close 43285 (time 1763335654.91 total 1062.84 last 78.88) - open/close 44111 (time 1763335664.91 total 1072.85 last 82.54) - open/close 44915 (time 1763335674.92 total 1082.85 last 80.37) - open/close 45725 (time 1763335684.93 total 1092.86 last 80.93) - open/close 46558 (time 1763335694.94 total 1102.88 last 83.17) - open/close 47340 (time 1763335704.96 total 1112.90 last 78.07) - open/close 48117 (time 1763335714.98 total 1122.91 last 77.56) - open/close 48901 (time 1763335724.98 total 1132.92 last 78.39) - open/close 49776 (time 1763335734.98 total 1142.92 last 87.43) total: 50000 open/close in 1145.60 seconds: 43.65 ops/second - unlinked 0 (time 1763335741 ; total 0 ; last 0) - unlinked 10000 (time 1763335869 ; total 128 ; last 128) - unlinked 20000 (time 1763335988 ; total 247 ; last 119) - unlinked 30000 (time 1763336111 ; total 370 ; last 123) - unlinked 40000 (time 1763336247 ; total 506 ; last 136) total: 50000 unlinks in 633 seconds: 78.988945 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local oleg249-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg249-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2