== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:04:43 (1769076283) start mds service on oleg131-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 117 (time 1769076370.45 total 10.03 last 11.67) - open/close 256 (time 1769076380.80 total 20.38 last 13.43) - open/close 332 (time 1769076390.83 total 30.40 last 7.58) - open/close 479 (time 1769076400.84 total 40.42 last 14.67) - open/close 617 (time 1769076411.50 total 51.08 last 12.95) - open/close 859 (time 1769076422.49 total 62.07 last 22.01) - open/close 1010 (time 1769076433.13 total 72.71 last 14.20) - open/close 1213 (time 1769076443.96 total 83.53 last 18.75) - open/close 1368 (time 1769076454.63 total 94.21 last 14.52) - open/close 1530 (time 1769076464.67 total 104.24 last 16.15) - open/close 1692 (time 1769076474.67 total 114.25 last 16.19) - open/close 1876 (time 1769076484.67 total 124.25 last 18.39) - open/close 1997 (time 1769076494.91 total 134.49 last 11.82) - open/close 2162 (time 1769076505.70 total 145.28 last 15.29) - open/close 2387 (time 1769076516.52 total 156.10 last 20.80) - open/close 2495 (time 1769076526.63 total 166.21 last 10.69) - open/close 2608 (time 1769076537.52 total 177.09 last 10.38) - open/close 2701 (time 1769076548.12 total 187.70 last 8.77) - open/close 2843 (time 1769076559.84 total 199.42 last 12.12) - open/close 3020 (time 1769076569.85 total 209.42 last 17.69) - open/close 3251 (time 1769076579.85 total 219.43 last 23.08) - open/close 3354 (time 1769076591.18 total 230.76 last 9.09) - open/close 3389 (time 1769076601.54 total 241.11 last 3.38) - open/close 3531 (time 1769076612.00 total 251.58 last 13.57) - open/close 3666 (time 1769076624.48 total 264.06 last 10.82) - open/close 3787 (time 1769076634.49 total 274.07 last 12.09) - open/close 3888 (time 1769076646.83 total 286.41 last 8.19) - open/close 4006 (time 1769076657.79 total 297.36 last 10.77) - open/close 4167 (time 1769076667.83 total 307.41 last 16.03) - open/close 4277 (time 1769076677.84 total 317.42 last 10.99) - open/close 4436 (time 1769076688.38 total 327.96 last 15.09) - open/close 4601 (time 1769076699.27 total 338.85 last 15.15) - open/close 4770 (time 1769076710.18 total 349.76 last 15.49) - open/close 4936 (time 1769076722.10 total 361.68 last 13.93) - open/close 5142 (time 1769076732.59 total 372.17 last 19.63) - open/close 5344 (time 1769076742.60 total 382.18 last 20.19) - open/close 5445 (time 1769076753.66 total 393.23 last 9.13) - open/close 5538 (time 1769076764.32 total 403.90 last 8.72) - open/close 5705 (time 1769076774.34 total 413.92 last 16.66) - open/close 5924 (time 1769076784.35 total 423.93 last 21.88) - open/close 6071 (time 1769076794.73 total 434.31 last 14.16) - open/close 6197 (time 1769076804.74 total 444.32 last 12.59) - open/close 6387 (time 1769076814.75 total 454.32 last 18.99) - open/close 6461 (time 1769076824.79 total 464.37 last 7.37) - open/close 6541 (time 1769076834.93 total 474.50 last 7.89) - open/close 6688 (time 1769076845.67 total 485.25 last 13.69) - open/close 6711 (time 1769076855.71 total 495.28 last 2.29) - open/close 6799 (time 1769076866.01 total 505.59 last 8.54) - open/close 6891 (time 1769076876.77 total 516.35 last 8.55) - open/close 6971 (time 1769076886.89 total 526.46 last 7.91) - open/close 7180 (time 1769076897.52 total 537.09 last 19.66) - open/close 7401 (time 1769076907.52 total 547.10 last 22.09) - open/close 7563 (time 1769076919.65 total 559.23 last 13.36) - open/close 7709 (time 1769076929.82 total 569.40 last 14.35) - open/close 7826 (time 1769076940.34 total 579.92 last 11.12) - open/close 7994 (time 1769076950.36 total 589.94 last 16.77) - open/close 8257 (time 1769076960.95 total 600.53 last 24.83) - open/close 8393 (time 1769076973.29 total 612.87 last 11.02) - open/close 8560 (time 1769076983.31 total 622.89 last 16.67) - open/close 8712 (time 1769076993.94 total 633.52 last 14.29) - open/close 8878 (time 1769077004.12 total 643.70 last 16.30) - open/close 9010 (time 1769077014.44 total 654.01 last 12.80) - open/close 9119 (time 1769077025.38 total 664.96 last 9.96) - open/close 9346 (time 1769077035.39 total 674.97 last 22.68) - open/close 9485 (time 1769077046.38 total 685.95 last 12.66) - open/close 9604 (time 1769077056.93 total 696.51 last 11.27) - open/close 9748 (time 1769077067.87 total 707.44 last 13.17) - open/close 9846 (time 1769077078.95 total 718.53 last 8.84) - open/close 10000 (time 1769077084.96 total 724.54 last 25.64) - open/close 10158 (time 1769077096.75 total 736.33 last 13.40) - open/close 10243 (time 1769077107.18 total 746.76 last 8.15) - open/close 10390 (time 1769077117.22 total 756.80 last 14.64) - open/close 10633 (time 1769077127.23 total 766.81 last 24.27) - open/close 10758 (time 1769077137.77 total 777.35 last 11.86) - open/close 10881 (time 1769077147.77 total 787.35 last 12.30) - open/close 11170 (time 1769077158.43 total 798.01 last 27.12) - open/close 11320 (time 1769077169.45 total 809.03 last 13.61) - open/close 11480 (time 1769077179.51 total 819.09 last 15.90) - open/close 11584 (time 1769077189.72 total 829.29 last 10.19) - open/close 11661 (time 1769077200.29 total 839.87 last 7.28) - open/close 11914 (time 1769077210.30 total 849.87 last 25.29) - open/close 12025 (time 1769077221.38 total 860.96 last 10.01) - open/close 12221 (time 1769077232.74 total 872.31 last 17.26) - open/close 12449 (time 1769077242.79 total 882.36 last 22.69) - open/close 12609 (time 1769077252.81 total 892.38 last 15.97) - open/close 12716 (time 1769077263.79 total 903.37 last 9.74) - open/close 12873 (time 1769077273.83 total 913.41 last 15.64) - open/close 13092 (time 1769077283.84 total 923.42 last 21.88) - open/close 13250 (time 1769077293.87 total 933.45 last 15.75) - open/close 13431 (time 1769077304.39 total 943.97 last 17.21) - open/close 13550 (time 1769077314.40 total 953.97 last 11.89) - open/close 13679 (time 1769077324.43 total 964.01 last 12.85) - open/close 13912 (time 1769077334.79 total 974.36 last 22.50) - open/close 14117 (time 1769077347.61 total 987.19 last 15.99) - open/close 14240 (time 1769077359.54 total 999.12 last 10.31) - open/close 14381 (time 1769077369.65 total 1009.23 last 13.95) - open/close 14464 (time 1769077380.61 total 1020.19 last 7.57) - open/close 14587 (time 1769077391.05 total 1030.63 last 11.79) - open/close 14734 (time 1769077401.72 total 1041.30 last 13.78) - open/close 14853 (time 1769077411.75 total 1051.33 last 11.86) - open/close 15032 (time 1769077422.48 total 1062.06 last 16.68) - open/close 15174 (time 1769077432.63 total 1072.20 last 14.00) - open/close 15271 (time 1769077442.63 total 1082.21 last 9.70) - open/close 15393 (time 1769077453.42 total 1092.99 last 11.31) - open/close 15491 (time 1769077463.48 total 1103.06 last 9.74) - open/close 15624 (time 1769077473.51 total 1113.09 last 13.26) - open/close 16114 (time 1769077483.54 total 1123.12 last 48.89) - open/close 16843 (time 1769077493.54 total 1133.12 last 72.89) - open/close 17713 (time 1769077503.54 total 1143.12 last 86.96) - open/close 18482 (time 1769077513.54 total 1153.12 last 76.89) - open/close 19355 (time 1769077523.55 total 1163.13 last 87.22) - open/close 20000 (time 1769077531.87 total 1171.45 last 77.58) - open/close 20866 (time 1769077541.87 total 1181.45 last 86.55) - open/close 21650 (time 1769077551.91 total 1191.48 last 78.14) - open/close 22382 (time 1769077561.91 total 1201.48 last 73.20) - open/close 23169 (time 1769077571.91 total 1211.49 last 78.64) - open/close 23875 (time 1769077581.92 total 1221.50 last 70.56) - open/close 24623 (time 1769077591.94 total 1231.51 last 74.68) - open/close 25369 (time 1769077601.95 total 1241.53 last 74.51) - open/close 26025 (time 1769077611.98 total 1251.55 last 65.42) - open/close 26734 (time 1769077621.99 total 1261.57 last 70.81) - open/close 27380 (time 1769077632.01 total 1271.59 last 64.46) - open/close 28089 (time 1769077642.02 total 1281.60 last 70.82) - open/close 28659 (time 1769077652.04 total 1291.61 last 56.92) - open/close 29190 (time 1769077662.04 total 1301.62 last 53.07) - open/close 29810 (time 1769077672.06 total 1311.63 last 61.92) - open/close 30000 (time 1769077675.20 total 1314.77 last 60.51) - open/close 30535 (time 1769077685.20 total 1324.78 last 53.48) - open/close 30983 (time 1769077695.21 total 1334.78 last 44.77) - open/close 31430 (time 1769077705.22 total 1344.79 last 44.65) - open/close 31852 (time 1769077715.22 total 1354.80 last 42.17) - open/close 32377 (time 1769077725.22 total 1364.80 last 52.50) - open/close 32936 (time 1769077735.25 total 1374.82 last 55.78) - open/close 33569 (time 1769077745.25 total 1384.83 last 63.28) - open/close 34250 (time 1769077755.26 total 1394.84 last 68.03) - open/close 34894 (time 1769077765.27 total 1404.85 last 64.34) - open/close 35543 (time 1769077775.27 total 1414.85 last 64.88) - open/close 36146 (time 1769077785.28 total 1424.86 last 60.24) - open/close 36738 (time 1769077795.30 total 1434.88 last 59.09) - open/close 37486 (time 1769077805.31 total 1444.89 last 74.74) - open/close 37985 (time 1769077815.33 total 1454.91 last 49.78) - open/close 38524 (time 1769077825.36 total 1464.93 last 53.77) - open/close 39148 (time 1769077835.36 total 1474.94 last 62.35) - open/close 39907 (time 1769077845.38 total 1484.95 last 75.80) - open/close 40697 (time 1769077855.38 total 1494.96 last 78.94) - open/close 41464 (time 1769077865.41 total 1504.98 last 76.53) - open/close 42157 (time 1769077875.41 total 1514.99 last 69.26) - open/close 42904 (time 1769077885.43 total 1525.00 last 74.60) - open/close 43520 (time 1769077895.44 total 1535.02 last 61.50) - open/close 44367 (time 1769077905.44 total 1545.02 last 84.69) - open/close 45077 (time 1769077915.44 total 1555.02 last 70.99) - open/close 45916 (time 1769077925.45 total 1565.03 last 83.83) - open/close 46555 (time 1769077935.46 total 1575.04 last 63.84) - open/close 47277 (time 1769077945.46 total 1585.04 last 72.20) - open/close 48066 (time 1769077955.47 total 1595.05 last 78.85) - open/close 48783 (time 1769077965.49 total 1605.06 last 71.58) - open/close 49493 (time 1769077975.57 total 1615.15 last 70.41) total: 50000 open/close in 1622.54 seconds: 30.82 ops/second - unlinked 0 (time 1769077986 ; total 0 ; last 0) - unlinked 10000 (time 1769078136 ; total 150 ; last 150) - unlinked 20000 (time 1769078282 ; total 296 ; last 146) - unlinked 30000 (time 1769078420 ; total 434 ; last 138) - unlinked 40000 (time 1769078551 ; total 565 ; last 131) total: 50000 unlinks in 686 seconds: 72.886299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg131-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2