== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:54:36 (1767318876) start mds service on oleg229-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1767318995.79 total 10.01 last 24.98) - open/close 497 (time 1767319005.80 total 20.01 last 24.68) - open/close 742 (time 1767319016.02 total 30.24 last 23.97) - open/close 976 (time 1767319026.06 total 40.27 last 23.33) - open/close 1282 (time 1767319036.70 total 50.91 last 28.75) - open/close 1505 (time 1767319047.39 total 61.60 last 20.87) - open/close 1736 (time 1767319057.41 total 71.62 last 23.05) - open/close 1964 (time 1767319068.36 total 82.57 last 20.82) - open/close 2211 (time 1767319078.36 total 92.57 last 24.70) - open/close 2390 (time 1767319088.52 total 102.73 last 17.62) - open/close 2669 (time 1767319098.52 total 112.73 last 27.89) - open/close 2855 (time 1767319108.74 total 122.96 last 18.19) - open/close 3081 (time 1767319119.43 total 133.64 last 21.15) - open/close 3245 (time 1767319129.99 total 144.20 last 15.53) - open/close 3477 (time 1767319140.03 total 154.24 last 23.11) - open/close 3658 (time 1767319150.21 total 164.42 last 17.77) - open/close 3854 (time 1767319160.28 total 174.49 last 19.47) - open/close 4077 (time 1767319171.10 total 185.31 last 20.61) - open/close 4331 (time 1767319181.10 total 195.31 last 25.39) - open/close 4555 (time 1767319191.77 total 205.98 last 21.00) - open/close 4735 (time 1767319201.79 total 216.00 last 17.96) - open/close 4980 (time 1767319212.81 total 227.02 last 22.23) - open/close 5220 (time 1767319223.11 total 237.33 last 23.30) - open/close 5465 (time 1767319233.86 total 248.07 last 22.80) - open/close 5683 (time 1767319243.91 total 258.12 last 21.70) - open/close 5891 (time 1767319254.79 total 269.00 last 19.11) - open/close 6100 (time 1767319264.81 total 279.02 last 20.86) - open/close 6310 (time 1767319275.52 total 289.74 last 19.60) - open/close 6552 (time 1767319285.53 total 299.75 last 24.18) - open/close 6776 (time 1767319298.51 total 312.72 last 17.27) - open/close 7020 (time 1767319308.51 total 322.72 last 24.39) - open/close 7248 (time 1767319318.51 total 332.72 last 22.79) - open/close 7506 (time 1767319328.58 total 342.79 last 25.64) - open/close 7760 (time 1767319339.03 total 353.24 last 24.30) - open/close 8002 (time 1767319349.06 total 363.27 last 24.12) - open/close 8238 (time 1767319359.06 total 373.28 last 23.59) - open/close 8460 (time 1767319369.28 total 383.50 last 21.72) - open/close 8787 (time 1767319379.79 total 394.01 last 31.11) - open/close 9075 (time 1767319389.80 total 404.01 last 28.78) - open/close 9359 (time 1767319399.80 total 414.02 last 28.39) - open/close 9531 (time 1767319409.89 total 424.10 last 17.06) - open/close 9759 (time 1767319419.89 total 434.10 last 22.80) - open/close 10000 (time 1767319428.43 total 442.65 last 28.20) - open/close 10227 (time 1767319439.19 total 453.41 last 21.10) - open/close 10436 (time 1767319449.20 total 463.41 last 20.88) - open/close 10680 (time 1767319459.21 total 473.43 last 24.37) - open/close 10917 (time 1767319469.33 total 483.54 last 23.44) - open/close 11135 (time 1767319480.33 total 494.54 last 19.82) - open/close 11367 (time 1767319490.33 total 504.54 last 23.20) - open/close 11666 (time 1767319500.34 total 514.55 last 29.86) - open/close 11845 (time 1767319510.86 total 525.07 last 17.02) - open/close 12137 (time 1767319521.80 total 536.02 last 26.69) - open/close 12363 (time 1767319531.81 total 546.02 last 22.59) - open/close 12579 (time 1767319541.83 total 556.04 last 21.56) - open/close 12800 (time 1767319551.83 total 566.05 last 22.09) - open/close 13053 (time 1767319562.52 total 576.73 last 23.68) - open/close 13324 (time 1767319572.53 total 586.74 last 27.08) - open/close 13565 (time 1767319582.92 total 597.14 last 23.18) - open/close 13848 (time 1767319593.94 total 608.15 last 25.70) - open/close 14048 (time 1767319604.82 total 619.03 last 18.38) - open/close 14304 (time 1767319614.82 total 629.03 last 25.60) - open/close 14534 (time 1767319624.83 total 639.05 last 22.97) - open/close 14689 (time 1767319634.83 total 649.05 last 15.50) - open/close 14833 (time 1767319644.84 total 659.05 last 14.40) - open/close 14994 (time 1767319655.56 total 669.77 last 15.01) - open/close 15193 (time 1767319665.56 total 679.78 last 19.90) - open/close 15443 (time 1767319676.17 total 690.38 last 23.58) - open/close 15674 (time 1767319687.04 total 701.26 last 21.24) - open/close 15928 (time 1767319697.87 total 712.08 last 23.47) - open/close 16162 (time 1767319707.88 total 722.09 last 23.37) - open/close 16381 (time 1767319718.66 total 732.87 last 20.32) - open/close 16622 (time 1767319729.36 total 743.57 last 22.53) - open/close 16924 (time 1767319739.36 total 753.57 last 30.20) - open/close 17126 (time 1767319749.37 total 763.58 last 20.18) - open/close 17352 (time 1767319759.86 total 774.07 last 21.54) - open/close 17593 (time 1767319770.88 total 785.10 last 21.86) - open/close 17822 (time 1767319781.06 total 795.27 last 22.51) - open/close 18053 (time 1767319791.19 total 805.40 last 22.80) - open/close 18313 (time 1767319801.62 total 815.83 last 24.92) - open/close 19273 (time 1767319811.63 total 825.84 last 95.94) - open/close 20000 (time 1767319821.41 total 835.63 last 74.27) - open/close 20991 (time 1767319831.42 total 845.63 last 99.05) - open/close 21859 (time 1767319841.44 total 855.65 last 86.64) - open/close 22652 (time 1767319851.44 total 865.66 last 79.26) - open/close 23539 (time 1767319861.45 total 875.66 last 88.65) - open/close 24384 (time 1767319871.45 total 885.66 last 84.48) - open/close 25242 (time 1767319881.46 total 895.67 last 85.73) - open/close 25938 (time 1767319891.47 total 905.68 last 69.54) - open/close 26753 (time 1767319901.47 total 915.68 last 81.48) - open/close 27607 (time 1767319911.48 total 925.70 last 85.31) - open/close 28592 (time 1767319921.49 total 935.70 last 98.45) - open/close 29432 (time 1767319931.50 total 945.71 last 83.92) - open/close 30000 (time 1767319938.44 total 952.65 last 81.81) - open/close 30879 (time 1767319948.44 total 962.66 last 87.85) - open/close 31757 (time 1767319958.45 total 972.66 last 87.76) - open/close 32547 (time 1767319968.47 total 982.68 last 78.85) - open/close 33394 (time 1767319978.47 total 992.69 last 84.66) - open/close 34309 (time 1767319988.48 total 1002.69 last 91.44) - open/close 35214 (time 1767319998.50 total 1012.71 last 90.30) - open/close 36108 (time 1767320008.52 total 1022.73 last 89.23) - open/close 36932 (time 1767320018.53 total 1032.75 last 82.30) - open/close 37942 (time 1767320028.54 total 1042.75 last 100.97) - open/close 38840 (time 1767320038.54 total 1052.75 last 89.76) - open/close 39733 (time 1767320048.55 total 1062.76 last 89.24) - open/close 40000 (time 1767320051.39 total 1065.60 last 93.95) - open/close 40846 (time 1767320061.39 total 1075.60 last 84.60) - open/close 41721 (time 1767320071.40 total 1085.61 last 87.44) - open/close 42648 (time 1767320081.40 total 1095.61 last 92.68) - open/close 43533 (time 1767320091.40 total 1105.61 last 88.47) - open/close 44432 (time 1767320101.41 total 1115.63 last 89.80) - open/close 45095 (time 1767320111.42 total 1125.63 last 66.28) - open/close 45883 (time 1767320121.42 total 1135.63 last 78.78) - open/close 46800 (time 1767320131.43 total 1145.64 last 91.59) - open/close 47916 (time 1767320141.44 total 1155.65 last 111.53) - open/close 48825 (time 1767320151.45 total 1165.66 last 90.82) - open/close 49782 (time 1767320161.45 total 1175.66 last 95.66) total: 50000 open/close in 1178.81 seconds: 42.42 ops/second - unlinked 0 (time 1767320168 ; total 0 ; last 0) - unlinked 10000 (time 1767320285 ; total 117 ; last 117) - unlinked 20000 (time 1767320378 ; total 210 ; last 93) - unlinked 30000 (time 1767320441 ; total 273 ; last 63) - unlinked 40000 (time 1767320507 ; total 339 ; last 66) total: 50000 unlinks in 412 seconds: 121.359222 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg229-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2