== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:46:05 (1743500765) start mds service on oleg641-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing load_modules_local oleg641-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg641-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg641-server: Loading modules from /home/green/git/lustre-release/lustre oleg641-server: detected 4 online CPUs by sysfs oleg641-server: MODOPTS_LIBCFS= oleg641-server: Force libcfs to create 2 CPU partitions oleg641-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg641-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg641-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1743500855.81 total 10.09 last 24.59) - open/close 432 (time 1743500868.04 total 22.32 last 15.04) - open/close 619 (time 1743500878.92 total 33.20 last 17.19) - open/close 867 (time 1743500888.93 total 43.21 last 24.78) - open/close 1138 (time 1743500899.78 total 54.06 last 24.97) - open/close 1369 (time 1743500910.59 total 64.87 last 21.38) - open/close 1443 (time 1743500920.59 total 74.87 last 7.40) - open/close 1612 (time 1743500930.69 total 84.97 last 16.72) - open/close 1789 (time 1743500941.32 total 95.61 last 16.65) - open/close 2095 (time 1743500951.33 total 105.61 last 30.59) - open/close 2212 (time 1743500962.95 total 117.23 last 10.07) - open/close 2261 (time 1743500973.50 total 127.78 last 4.64) - open/close 2349 (time 1743500983.69 total 137.97 last 8.64) - open/close 2435 (time 1743500994.75 total 149.03 last 7.78) - open/close 2576 (time 1743501007.24 total 161.52 last 11.28) - open/close 2716 (time 1743501018.51 total 172.79 last 12.42) - open/close 2919 (time 1743501028.53 total 182.81 last 20.27) - open/close 3059 (time 1743501038.53 total 192.81 last 13.99) - open/close 3143 (time 1743501049.61 total 203.89 last 7.59) - open/close 3229 (time 1743501059.78 total 214.06 last 8.46) - open/close 3503 (time 1743501069.79 total 224.07 last 27.37) - open/close 3656 (time 1743501080.32 total 234.60 last 14.53) - open/close 3777 (time 1743501093.19 total 247.47 last 9.40) - open/close 3970 (time 1743501104.07 total 258.35 last 17.73) - open/close 4015 (time 1743501115.45 total 269.73 last 3.96) - open/close 4086 (time 1743501127.48 total 281.77 last 5.90) - open/close 4140 (time 1743501140.93 total 295.21 last 4.02) - open/close 4324 (time 1743501150.94 total 305.22 last 18.38) - open/close 4564 (time 1743501161.03 total 315.31 last 23.78) - open/close 4738 (time 1743501171.77 total 326.05 last 16.21) - open/close 5050 (time 1743501181.78 total 336.06 last 31.15) - open/close 5355 (time 1743501192.19 total 346.47 last 29.31) - open/close 5585 (time 1743501203.90 total 358.18 last 19.64) - open/close 5847 (time 1743501213.91 total 368.19 last 26.16) - open/close 6128 (time 1743501224.38 total 378.66 last 26.84) - open/close 6437 (time 1743501234.56 total 388.84 last 30.36) - open/close 6720 (time 1743501244.57 total 398.85 last 28.28) - open/close 6958 (time 1743501254.57 total 408.85 last 23.80) - open/close 7272 (time 1743501264.58 total 418.86 last 31.37) - open/close 7567 (time 1743501274.59 total 428.87 last 29.47) - open/close 7832 (time 1743501284.59 total 438.88 last 26.48) - open/close 8089 (time 1743501294.60 total 448.88 last 25.68) - open/close 8258 (time 1743501305.40 total 459.68 last 15.65) - open/close 8350 (time 1743501315.46 total 469.74 last 9.15) - open/close 8567 (time 1743501325.48 total 479.76 last 21.66) - open/close 8742 (time 1743501335.62 total 489.90 last 17.26) - open/close 9052 (time 1743501346.19 total 500.47 last 29.33) - open/close 9376 (time 1743501356.47 total 510.75 last 31.50) - open/close 9644 (time 1743501366.48 total 520.76 last 26.79) - open/close 9866 (time 1743501376.48 total 530.76 last 22.20) - open/close 10000 (time 1743501380.74 total 535.03 last 31.40) - open/close 10131 (time 1743501391.80 total 546.08 last 11.85) - open/close 10375 (time 1743501402.37 total 556.65 last 23.09) - open/close 10619 (time 1743501412.38 total 566.66 last 24.37) - open/close 10812 (time 1743501422.40 total 576.68 last 19.27) - open/close 11093 (time 1743501432.41 total 586.69 last 28.06) - open/close 11316 (time 1743501442.57 total 596.85 last 21.94) - open/close 11548 (time 1743501452.59 total 606.87 last 23.17) - open/close 11777 (time 1743501462.59 total 616.87 last 22.90) - open/close 12060 (time 1743501474.30 total 628.58 last 24.16) - open/close 12359 (time 1743501484.41 total 638.69 last 29.58) - open/close 12603 (time 1743501495.11 total 649.39 last 22.80) - open/close 12924 (time 1743501505.12 total 659.40 last 32.06) - open/close 13283 (time 1743501515.13 total 669.41 last 35.88) - open/close 13584 (time 1743501525.14 total 679.42 last 30.05) - open/close 13966 (time 1743501535.15 total 689.44 last 38.16) - open/close 14298 (time 1743501545.84 total 700.12 last 31.07) - open/close 14643 (time 1743501555.84 total 710.12 last 34.49) - open/close 15016 (time 1743501566.91 total 721.19 last 33.71) - open/close 15411 (time 1743501580.00 total 734.28 last 30.17) - open/close 15805 (time 1743501590.00 total 744.28 last 39.38) - open/close 16124 (time 1743501600.01 total 754.29 last 31.86) - open/close 16468 (time 1743501610.04 total 764.32 last 34.31) - open/close 16798 (time 1743501620.04 total 774.32 last 33.00) - open/close 17068 (time 1743501630.06 total 784.34 last 26.96) - open/close 17381 (time 1743501640.44 total 794.72 last 30.15) - open/close 17711 (time 1743501650.44 total 804.73 last 32.98) - open/close 18092 (time 1743501660.92 total 815.20 last 36.38) - open/close 18485 (time 1743501670.92 total 825.20 last 39.28) - open/close 18851 (time 1743501681.52 total 835.80 last 34.53) - open/close 19236 (time 1743501691.53 total 845.81 last 38.45) - open/close 19608 (time 1743501702.61 total 856.90 last 33.57) - open/close 20000 (time 1743501712.18 total 866.46 last 40.97) - open/close 20385 (time 1743501722.19 total 876.47 last 38.47) - open/close 20734 (time 1743501732.19 total 886.48 last 34.88) - open/close 21067 (time 1743501742.64 total 896.92 last 31.88) - open/close 21497 (time 1743501752.66 total 906.94 last 42.93) - open/close 21852 (time 1743501763.25 total 917.53 last 33.51) - open/close 22176 (time 1743501773.26 total 927.54 last 32.38) - open/close 22513 (time 1743501783.80 total 938.08 last 31.98) - open/close 22899 (time 1743501794.85 total 949.13 last 34.93) - open/close 23271 (time 1743501804.86 total 959.14 last 37.14) - open/close 23601 (time 1743501815.37 total 969.65 last 31.41) - open/close 23976 (time 1743501825.46 total 979.74 last 37.18) - open/close 24218 (time 1743501836.26 total 990.54 last 22.41) - open/close 24499 (time 1743501846.49 total 1000.77 last 27.46) - open/close 24850 (time 1743501856.50 total 1010.78 last 35.06) - open/close 25210 (time 1743501866.51 total 1020.79 last 35.96) - open/close 25588 (time 1743501876.52 total 1030.80 last 37.77) - open/close 25989 (time 1743501886.53 total 1040.81 last 40.08) - open/close 27011 (time 1743501896.53 total 1050.81 last 102.19) - open/close 28120 (time 1743501906.53 total 1060.81 last 110.89) - open/close 29170 (time 1743501916.53 total 1070.81 last 104.99) - open/close 30000 (time 1743501924.77 total 1079.05 last 100.69) - open/close 31011 (time 1743501934.77 total 1089.05 last 101.10) - open/close 31885 (time 1743501944.78 total 1099.06 last 87.37) - open/close 32841 (time 1743501954.78 total 1109.06 last 95.59) - open/close 33856 (time 1743501964.78 total 1119.06 last 101.48) - open/close 34772 (time 1743501974.78 total 1129.06 last 91.57) - open/close 35655 (time 1743501984.79 total 1139.07 last 88.27) - open/close 36691 (time 1743501994.79 total 1149.07 last 103.56) - open/close 37705 (time 1743502004.88 total 1159.16 last 100.52) - open/close 38693 (time 1743502014.90 total 1169.18 last 98.57) - open/close 39606 (time 1743502024.91 total 1179.19 last 91.23) - open/close 40000 (time 1743502029.38 total 1183.66 last 88.19) - open/close 40902 (time 1743502039.38 total 1193.66 last 90.18) - open/close 41859 (time 1743502049.38 total 1203.66 last 95.69) - open/close 42692 (time 1743502059.38 total 1213.66 last 83.26) - open/close 43573 (time 1743502069.38 total 1223.67 last 88.09) - open/close 44468 (time 1743502079.39 total 1233.67 last 89.47) - open/close 45451 (time 1743502089.39 total 1243.67 last 98.30) - open/close 46589 (time 1743502099.41 total 1253.70 last 113.51) - open/close 47668 (time 1743502109.42 total 1263.70 last 107.89) - open/close 48614 (time 1743502119.42 total 1273.70 last 94.59) - open/close 49504 (time 1743502129.42 total 1283.71 last 88.93) total: 50000 open/close in 1289.22 seconds: 38.78 ops/second - unlinked 0 (time 1743502140 ; total 0 ; last 0) - unlinked 10000 (time 1743502274 ; total 134 ; last 134) - unlinked 20000 (time 1743502408 ; total 268 ; last 134) - unlinked 30000 (time 1743502546 ; total 406 ; last 138) - unlinked 40000 (time 1743502673 ; total 533 ; last 127) total: 50000 unlinks in 651 seconds: 76.804916 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg641-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing unload_modules_local oleg641-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg641-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2