== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:37:15 (1786822635) start mds service on oleg208-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg208-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 141 (time 1786822729.09 total 10.80 last 13.06) - open/close 209 (time 1786822739.14 total 20.84 last 6.77) - open/close 336 (time 1786822749.55 total 31.25 last 12.20) - open/close 393 (time 1786822760.39 total 42.09 last 5.26) - open/close 583 (time 1786822770.41 total 52.11 last 18.96) - open/close 749 (time 1786822780.43 total 62.13 last 16.57) - open/close 907 (time 1786822791.43 total 73.13 last 14.36) - open/close 1085 (time 1786822801.54 total 83.24 last 17.61) - open/close 1257 (time 1786822811.55 total 93.25 last 17.18) - open/close 1461 (time 1786822821.56 total 103.26 last 20.38) - open/close 1691 (time 1786822831.91 total 113.61 last 22.22) - open/close 1859 (time 1786822842.90 total 124.61 last 15.28) - open/close 2028 (time 1786822854.99 total 136.70 last 13.98) - open/close 2207 (time 1786822865.11 total 146.81 last 17.69) - open/close 2411 (time 1786822875.40 total 157.10 last 19.82) - open/close 2611 (time 1786822885.40 total 167.11 last 20.00) - open/close 2838 (time 1786822896.33 total 178.03 last 20.78) - open/close 3062 (time 1786822907.17 total 188.87 last 20.66) - open/close 3232 (time 1786822917.95 total 199.65 last 15.77) - open/close 3403 (time 1786822927.98 total 209.68 last 17.05) - open/close 3618 (time 1786822938.51 total 220.22 last 20.41) - open/close 3827 (time 1786822948.57 total 230.27 last 20.79) - open/close 3962 (time 1786822958.97 total 240.67 last 12.98) - open/close 4084 (time 1786822968.98 total 250.68 last 12.19) - open/close 4275 (time 1786822979.27 total 260.97 last 18.57) - open/close 4488 (time 1786822989.27 total 270.97 last 21.29) - open/close 4703 (time 1786823001.70 total 283.40 last 17.30) - open/close 4877 (time 1786823012.22 total 293.93 last 16.53) - open/close 5043 (time 1786823022.22 total 303.93 last 16.60) - open/close 5166 (time 1786823032.49 total 314.19 last 11.98) - open/close 5378 (time 1786823042.49 total 324.19 last 21.19) - open/close 5538 (time 1786823053.32 total 335.02 last 14.78) - open/close 5651 (time 1786823063.34 total 345.04 last 11.28) - open/close 5860 (time 1786823073.38 total 355.08 last 20.81) - open/close 6066 (time 1786823083.58 total 365.28 last 20.20) - open/close 6319 (time 1786823094.09 total 375.79 last 24.07) - open/close 6435 (time 1786823104.51 total 386.21 last 11.14) - open/close 6461 (time 1786823114.54 total 396.24 last 2.59) - open/close 6588 (time 1786823124.56 total 406.27 last 12.67) - open/close 6751 (time 1786823134.57 total 416.27 last 16.30) - open/close 6990 (time 1786823144.57 total 426.27 last 23.89) - open/close 7172 (time 1786823155.02 total 436.72 last 17.42) - open/close 7294 (time 1786823165.03 total 446.73 last 12.18) - open/close 7376 (time 1786823175.05 total 456.75 last 8.18) - open/close 7534 (time 1786823185.06 total 466.76 last 15.79) - open/close 7629 (time 1786823195.06 total 476.76 last 9.50) - open/close 7790 (time 1786823205.13 total 486.83 last 15.98) - open/close 7933 (time 1786823215.30 total 497.00 last 14.07) - open/close 8083 (time 1786823228.79 total 510.50 last 11.11) - open/close 8272 (time 1786823238.82 total 520.53 last 18.84) - open/close 8467 (time 1786823249.03 total 530.73 last 19.11) - open/close 8647 (time 1786823263.50 total 545.20 last 12.44) - open/close 8766 (time 1786823273.64 total 555.34 last 11.73) - open/close 9008 (time 1786823283.70 total 565.40 last 24.07) - open/close 9210 (time 1786823293.70 total 575.40 last 20.19) - open/close 9365 (time 1786823303.71 total 585.42 last 15.48) - open/close 9508 (time 1786823314.06 total 595.76 last 13.83) - open/close 9594 (time 1786823325.00 total 606.71 last 7.85) - open/close 9686 (time 1786823335.02 total 616.72 last 9.19) - open/close 9900 (time 1786823345.03 total 626.73 last 21.38) - open/close 10000 (time 1786823352.32 total 634.02 last 13.72) - open/close 10227 (time 1786823362.32 total 644.02 last 22.70) - open/close 10415 (time 1786823374.78 total 656.48 last 15.09) - open/close 10583 (time 1786823385.61 total 667.31 last 15.52) - open/close 10777 (time 1786823396.44 total 678.15 last 17.90) - open/close 10908 (time 1786823406.61 total 688.31 last 12.89) - open/close 11112 (time 1786823417.08 total 698.79 last 19.48) - open/close 11342 (time 1786823427.09 total 708.79 last 22.98) - open/close 11526 (time 1786823438.28 total 719.98 last 16.45) - open/close 11808 (time 1786823448.28 total 729.98 last 28.20) - open/close 12057 (time 1786823459.38 total 741.08 last 22.44) - open/close 12291 (time 1786823470.36 total 752.06 last 21.31) - open/close 12434 (time 1786823482.20 total 763.90 last 12.07) - open/close 12552 (time 1786823494.40 total 776.10 last 9.67) - open/close 12662 (time 1786823508.25 total 789.95 last 7.95) - open/close 12790 (time 1786823522.62 total 804.32 last 8.91) - open/close 12906 (time 1786823532.70 total 814.40 last 11.51) - open/close 13096 (time 1786823542.72 total 824.42 last 18.96) - open/close 13332 (time 1786823553.29 total 834.99 last 22.32) - open/close 13450 (time 1786823563.29 total 845.00 last 11.80) - open/close 13717 (time 1786823573.63 total 855.33 last 25.83) - open/close 13951 (time 1786823583.66 total 865.36 last 23.34) - open/close 14186 (time 1786823594.63 total 876.34 last 21.41) - open/close 14401 (time 1786823605.27 total 886.98 last 20.21) - open/close 14658 (time 1786823615.87 total 897.57 last 24.26) - open/close 14878 (time 1786823627.88 total 909.58 last 18.31) - open/close 14958 (time 1786823637.89 total 919.59 last 7.99) - open/close 15164 (time 1786823648.33 total 930.03 last 19.74) - open/close 15317 (time 1786823658.57 total 940.27 last 14.94) - open/close 15422 (time 1786823670.16 total 951.86 last 9.06) - open/close 15529 (time 1786823680.90 total 962.60 last 9.96) - open/close 15682 (time 1786823691.77 total 973.47 last 14.07) - open/close 15917 (time 1786823702.27 total 983.97 last 22.38) - open/close 16064 (time 1786823712.28 total 993.98 last 14.69) - open/close 16292 (time 1786823723.31 total 1005.01 last 20.67) - open/close 16538 (time 1786823733.32 total 1015.02 last 24.58) - open/close 16724 (time 1786823744.34 total 1026.04 last 16.88) - open/close 16821 (time 1786823754.69 total 1036.39 last 9.37) - open/close 17514 (time 1786823764.70 total 1046.40 last 69.24) - open/close 18219 (time 1786823774.70 total 1056.40 last 70.48) - open/close 18918 (time 1786823784.71 total 1066.41 last 69.85) - open/close 19607 (time 1786823794.72 total 1076.42 last 68.81) - open/close 20000 (time 1786823800.86 total 1082.57 last 63.96) - open/close 20761 (time 1786823810.87 total 1092.57 last 76.10) - open/close 21488 (time 1786823820.87 total 1102.57 last 72.68) - open/close 22145 (time 1786823830.88 total 1112.58 last 65.64) - open/close 22852 (time 1786823840.88 total 1122.58 last 70.69) - open/close 23407 (time 1786823850.88 total 1132.59 last 55.47) - open/close 23915 (time 1786823860.89 total 1142.59 last 50.79) - open/close 24426 (time 1786823870.89 total 1152.59 last 51.06) - open/close 25137 (time 1786823880.90 total 1162.60 last 71.06) - open/close 25865 (time 1786823890.90 total 1172.61 last 72.76) - open/close 26608 (time 1786823900.92 total 1182.62 last 74.16) - open/close 27293 (time 1786823910.92 total 1192.63 last 68.49) - open/close 27961 (time 1786823920.93 total 1202.63 last 66.79) - open/close 28598 (time 1786823930.93 total 1212.63 last 63.65) - open/close 29236 (time 1786823940.95 total 1222.65 last 63.72) - open/close 29859 (time 1786823950.96 total 1232.66 last 62.24) - open/close 30537 (time 1786823960.96 total 1242.67 last 67.75) - open/close 31237 (time 1786823970.97 total 1252.67 last 69.93) - open/close 31934 (time 1786823980.98 total 1262.68 last 69.63) - open/close 32597 (time 1786823990.99 total 1272.70 last 66.23) - open/close 33251 (time 1786824001.00 total 1282.70 last 65.34) - open/close 33870 (time 1786824011.01 total 1292.71 last 61.85) - open/close 34548 (time 1786824021.02 total 1302.72 last 67.75) - open/close 35321 (time 1786824031.02 total 1312.72 last 77.30) - open/close 36012 (time 1786824041.02 total 1322.72 last 69.09) - open/close 36708 (time 1786824051.03 total 1332.74 last 69.50) - open/close 37441 (time 1786824061.06 total 1342.76 last 73.09) - open/close 38037 (time 1786824071.07 total 1352.77 last 59.57) - open/close 38735 (time 1786824081.07 total 1362.77 last 69.76) - open/close 39374 (time 1786824091.08 total 1372.78 last 63.86) - open/close 40000 (time 1786824100.03 total 1381.73 last 69.97) - open/close 40769 (time 1786824110.03 total 1391.73 last 76.87) - open/close 41510 (time 1786824120.04 total 1401.74 last 74.03) - open/close 42109 (time 1786824130.06 total 1411.76 last 59.79) - open/close 42766 (time 1786824140.08 total 1421.78 last 65.57) - open/close 43346 (time 1786824150.09 total 1431.79 last 57.92) - open/close 43831 (time 1786824160.11 total 1441.81 last 48.40) - open/close 44434 (time 1786824170.12 total 1451.82 last 60.27) - open/close 45105 (time 1786824180.13 total 1461.84 last 66.99) - open/close 45695 (time 1786824190.17 total 1471.88 last 58.77) - open/close 46187 (time 1786824200.19 total 1481.89 last 49.14) - open/close 46850 (time 1786824210.20 total 1491.90 last 66.21) - open/close 47417 (time 1786824220.21 total 1501.91 last 56.66) - open/close 48028 (time 1786824230.21 total 1511.91 last 61.08) - open/close 48596 (time 1786824240.22 total 1521.92 last 56.75) - open/close 49191 (time 1786824250.23 total 1531.93 last 59.45) - open/close 49750 (time 1786824260.24 total 1541.94 last 55.82) total: 50000 open/close in 1545.86 seconds: 32.34 ops/second - unlinked 0 (time 1786824268 ; total 0 ; last 0) - unlinked 10000 (time 1786824434 ; total 166 ; last 166) - unlinked 20000 (time 1786824596 ; total 328 ; last 162) - unlinked 30000 (time 1786824756 ; total 488 ; last 160) - unlinked 40000 (time 1786824918 ; total 650 ; last 162) total: 50000 unlinks in 813 seconds: 61.500614 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2