== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:34 (1769057434) start mds service on oleg326-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions oleg326-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg326-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg326-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1769057505.58 total 10.00 last 24.80) - open/close 456 (time 1769057516.04 total 20.46 last 19.89) - open/close 588 (time 1769057526.05 total 30.47 last 13.19) - open/close 706 (time 1769057536.09 total 40.51 last 11.75) - open/close 818 (time 1769057547.01 total 51.43 last 10.25) - open/close 951 (time 1769057557.08 total 61.50 last 13.21) - open/close 1041 (time 1769057567.10 total 71.52 last 8.99) - open/close 1138 (time 1769057577.99 total 82.41 last 8.90) - open/close 1265 (time 1769057588.15 total 92.57 last 12.50) - open/close 1410 (time 1769057598.53 total 102.95 last 13.97) - open/close 1518 (time 1769057609.34 total 113.76 last 9.99) - open/close 1694 (time 1769057619.36 total 123.78 last 17.57) - open/close 1858 (time 1769057630.14 total 134.56 last 15.22) - open/close 1990 (time 1769057640.19 total 144.61 last 13.13) - open/close 2143 (time 1769057650.70 total 155.12 last 14.56) - open/close 2297 (time 1769057661.34 total 165.76 last 14.46) - open/close 2455 (time 1769057672.08 total 176.50 last 14.71) - open/close 2571 (time 1769057682.20 total 186.62 last 11.47) - open/close 2667 (time 1769057693.06 total 197.48 last 8.83) - open/close 2756 (time 1769057703.55 total 207.97 last 8.48) - open/close 2849 (time 1769057714.56 total 218.98 last 8.45) - open/close 2921 (time 1769057725.19 total 229.61 last 6.77) - open/close 3051 (time 1769057735.60 total 240.02 last 12.48) - open/close 3210 (time 1769057746.25 total 250.67 last 14.94) - open/close 3377 (time 1769057756.99 total 261.41 last 15.55) - open/close 3449 (time 1769057767.00 total 271.42 last 7.19) - open/close 3559 (time 1769057777.05 total 281.47 last 10.95) - open/close 3747 (time 1769057787.05 total 291.47 last 18.79) - open/close 3930 (time 1769057797.96 total 302.38 last 16.78) - open/close 4127 (time 1769057807.98 total 312.40 last 19.66) - open/close 4330 (time 1769057818.02 total 322.44 last 20.21) - open/close 4439 (time 1769057828.03 total 332.45 last 10.89) - open/close 4504 (time 1769057838.09 total 342.52 last 6.46) - open/close 4674 (time 1769057849.04 total 353.46 last 15.53) - open/close 4802 (time 1769057859.67 total 364.09 last 12.05) - open/close 4947 (time 1769057872.90 total 377.32 last 10.96) - open/close 5121 (time 1769057882.91 total 387.33 last 17.38) - open/close 5337 (time 1769057893.39 total 397.81 last 20.61) - open/close 5457 (time 1769057904.13 total 408.55 last 11.17) - open/close 5658 (time 1769057914.39 total 418.81 last 19.59) - open/close 5782 (time 1769057924.41 total 428.83 last 12.38) - open/close 5914 (time 1769057934.45 total 438.87 last 13.15) - open/close 6020 (time 1769057944.46 total 448.88 last 10.59) - open/close 6201 (time 1769057954.47 total 458.89 last 18.08) - open/close 6472 (time 1769057964.61 total 469.03 last 26.72) - open/close 6605 (time 1769057974.86 total 479.28 last 12.99) - open/close 6676 (time 1769057984.87 total 489.29 last 7.09) - open/close 6847 (time 1769057994.87 total 499.29 last 17.09) - open/close 7098 (time 1769058004.89 total 509.31 last 25.05) - open/close 7205 (time 1769058015.78 total 520.20 last 9.83) - open/close 7324 (time 1769058025.78 total 530.20 last 11.90) - open/close 7529 (time 1769058036.87 total 541.29 last 18.49) - open/close 7666 (time 1769058046.90 total 551.32 last 13.66) - open/close 7856 (time 1769058056.92 total 561.34 last 18.95) - open/close 8078 (time 1769058070.04 total 574.46 last 16.92) - open/close 8276 (time 1769058080.05 total 584.47 last 19.80) - open/close 8415 (time 1769058090.05 total 594.47 last 13.89) - open/close 8581 (time 1769058100.06 total 604.48 last 16.59) - open/close 8746 (time 1769058110.55 total 614.97 last 15.73) - open/close 8920 (time 1769058120.56 total 624.98 last 17.38) - open/close 9054 (time 1769058131.49 total 635.91 last 12.27) - open/close 9174 (time 1769058141.69 total 646.11 last 11.76) - open/close 9359 (time 1769058151.99 total 656.42 last 17.96) - open/close 9485 (time 1769058162.01 total 666.43 last 12.58) - open/close 9619 (time 1769058172.11 total 676.53 last 13.27) - open/close 9805 (time 1769058182.59 total 687.01 last 17.74) - open/close 10000 (time 1769058190.87 total 695.29 last 23.55) - open/close 10079 (time 1769058200.90 total 705.32 last 7.88) - open/close 10260 (time 1769058210.90 total 715.32 last 18.09) - open/close 10380 (time 1769058220.92 total 725.34 last 11.97) - open/close 10536 (time 1769058231.46 total 735.88 last 14.80) - open/close 10664 (time 1769058241.48 total 745.90 last 12.77) - open/close 10848 (time 1769058251.93 total 756.35 last 17.61) - open/close 11070 (time 1769058262.74 total 767.16 last 20.53) - open/close 11267 (time 1769058272.74 total 777.17 last 19.69) - open/close 11396 (time 1769058282.95 total 787.37 last 12.64) - open/close 11617 (time 1769058294.92 total 799.34 last 18.47) - open/close 11855 (time 1769058304.92 total 809.34 last 23.78) - open/close 11927 (time 1769058315.14 total 819.57 last 7.04) - open/close 12193 (time 1769058325.96 total 830.38 last 24.60) - open/close 12356 (time 1769058336.83 total 841.25 last 15.00) - open/close 12535 (time 1769058347.46 total 851.88 last 16.83) - open/close 12732 (time 1769058358.08 total 862.50 last 18.55) - open/close 12986 (time 1769058369.15 total 873.57 last 22.95) - open/close 13193 (time 1769058379.20 total 883.62 last 20.61) - open/close 13452 (time 1769058390.17 total 894.59 last 23.61) - open/close 13714 (time 1769058400.91 total 905.33 last 24.38) - open/close 13845 (time 1769058414.04 total 918.47 last 9.98) - open/close 13973 (time 1769058424.05 total 928.47 last 12.79) - open/close 14137 (time 1769058434.39 total 938.81 last 15.86) - open/close 14361 (time 1769058444.40 total 948.82 last 22.38) - open/close 14482 (time 1769058456.26 total 960.68 last 10.20) - open/close 14762 (time 1769058466.30 total 970.72 last 27.90) - open/close 14887 (time 1769058476.54 total 980.96 last 12.20) - open/close 15087 (time 1769058486.55 total 990.97 last 19.98) - open/close 15313 (time 1769058497.42 total 1001.84 last 20.78) - open/close 15537 (time 1769058508.38 total 1012.80 last 20.45) - open/close 15763 (time 1769058520.90 total 1025.32 last 18.05) - open/close 15953 (time 1769058530.90 total 1035.32 last 18.99) - open/close 16255 (time 1769058541.63 total 1046.05 last 28.15) - open/close 16480 (time 1769058551.99 total 1056.41 last 21.72) - open/close 16680 (time 1769058563.04 total 1067.46 last 18.11) - open/close 17448 (time 1769058573.04 total 1077.46 last 76.74) - open/close 18181 (time 1769058583.05 total 1087.48 last 73.22) - open/close 18934 (time 1769058593.06 total 1097.48 last 75.25) - open/close 19717 (time 1769058603.07 total 1107.49 last 78.25) - open/close 20000 (time 1769058606.75 total 1111.17 last 76.86) - open/close 20924 (time 1769058616.76 total 1121.18 last 92.31) - open/close 21850 (time 1769058626.76 total 1131.18 last 92.57) - open/close 22640 (time 1769058636.80 total 1141.22 last 78.70) - open/close 23382 (time 1769058646.81 total 1151.23 last 74.13) - open/close 24141 (time 1769058656.82 total 1161.24 last 75.86) - open/close 24913 (time 1769058666.83 total 1171.25 last 77.12) - open/close 25756 (time 1769058676.83 total 1181.25 last 84.27) - open/close 26636 (time 1769058686.83 total 1191.25 last 87.98) - open/close 27424 (time 1769058696.83 total 1201.26 last 78.78) - open/close 28157 (time 1769058706.84 total 1211.26 last 73.29) - open/close 28990 (time 1769058716.84 total 1221.26 last 83.29) - open/close 29743 (time 1769058726.85 total 1231.27 last 75.20) - open/close 30000 (time 1769058730.76 total 1235.18 last 65.81) - open/close 30817 (time 1769058740.76 total 1245.18 last 81.69) - open/close 31724 (time 1769058750.76 total 1255.18 last 90.66) - open/close 32495 (time 1769058760.76 total 1265.18 last 77.09) - open/close 33296 (time 1769058770.77 total 1275.19 last 80.03) - open/close 34108 (time 1769058780.77 total 1285.19 last 81.19) - open/close 35005 (time 1769058790.78 total 1295.20 last 89.65) - open/close 35868 (time 1769058800.78 total 1305.20 last 86.26) - open/close 36727 (time 1769058810.79 total 1315.22 last 85.81) - open/close 37597 (time 1769058820.80 total 1325.22 last 86.95) - open/close 38490 (time 1769058830.80 total 1335.22 last 89.27) - open/close 39295 (time 1769058840.81 total 1345.23 last 80.43) - open/close 40000 (time 1769058849.25 total 1353.67 last 83.57) - open/close 40750 (time 1769058859.25 total 1363.67 last 74.99) - open/close 41524 (time 1769058869.26 total 1373.68 last 77.33) - open/close 42339 (time 1769058879.29 total 1383.71 last 81.27) - open/close 42994 (time 1769058889.29 total 1393.71 last 65.47) - open/close 43782 (time 1769058899.30 total 1403.72 last 78.76) - open/close 44592 (time 1769058909.30 total 1413.72 last 80.95) - open/close 45374 (time 1769058919.31 total 1423.73 last 78.14) - open/close 46229 (time 1769058929.31 total 1433.73 last 85.46) - open/close 47010 (time 1769058939.32 total 1443.74 last 78.06) - open/close 47862 (time 1769058949.33 total 1453.75 last 85.14) - open/close 48720 (time 1769058959.34 total 1463.76 last 85.71) - open/close 49594 (time 1769058969.35 total 1473.77 last 87.24) total: 50000 open/close in 1478.31 seconds: 33.82 ops/second - unlinked 0 (time 1769058978 ; total 0 ; last 0) - unlinked 10000 (time 1769059105 ; total 127 ; last 127) - unlinked 20000 (time 1769059230 ; total 252 ; last 125) - unlinked 30000 (time 1769059346 ; total 368 ; last 116) - unlinked 40000 (time 1769059463 ; total 485 ; last 117) total: 50000 unlinks in 604 seconds: 82.781456 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local oleg326-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg326-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2