== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:56:16 (1709787376) start mds service on oleg205-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg205-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 184 (time 1709787428.64 total 10.78 last 17.06) - open/close 439 (time 1709787438.65 total 20.79 last 25.47) - open/close 622 (time 1709787448.91 total 31.05 last 17.84) - open/close 805 (time 1709787458.99 total 41.13 last 18.16) - open/close 1105 (time 1709787469.03 total 51.17 last 29.88) - open/close 1265 (time 1709787479.81 total 61.95 last 14.84) - open/close 1463 (time 1709787490.80 total 72.94 last 18.01) - open/close 1744 (time 1709787501.16 total 83.30 last 27.11) - open/close 1917 (time 1709787512.00 total 94.14 last 15.97) - open/close 2091 (time 1709787522.94 total 105.08 last 15.90) - open/close 2312 (time 1709787533.56 total 115.70 last 20.82) - open/close 2573 (time 1709787543.80 total 125.94 last 25.50) - open/close 2712 (time 1709787553.80 total 135.95 last 13.89) - open/close 2935 (time 1709787563.81 total 145.95 last 22.29) - open/close 3074 (time 1709787574.02 total 156.16 last 13.62) - open/close 3240 (time 1709787584.05 total 166.19 last 16.55) - open/close 3424 (time 1709787594.07 total 176.21 last 18.36) - open/close 3578 (time 1709787605.10 total 187.24 last 13.96) - open/close 3700 (time 1709787615.56 total 197.70 last 11.66) - open/close 3955 (time 1709787626.51 total 208.65 last 23.30) - open/close 4127 (time 1709787636.83 total 218.97 last 16.66) - open/close 4251 (time 1709787647.29 total 229.43 last 11.86) - open/close 4426 (time 1709787657.89 total 240.04 last 16.50) - open/close 4543 (time 1709787668.93 total 251.07 last 10.60) - open/close 4706 (time 1709787678.93 total 261.08 last 16.30) - open/close 4928 (time 1709787689.30 total 271.44 last 21.42) - open/close 5130 (time 1709787699.53 total 281.68 last 19.73) - open/close 5259 (time 1709787710.15 total 292.29 last 12.15) - open/close 5393 (time 1709787720.18 total 302.32 last 13.36) - open/close 5538 (time 1709787730.52 total 312.66 last 14.02) - open/close 5750 (time 1709787741.09 total 323.23 last 20.06) - open/close 5896 (time 1709787751.11 total 333.25 last 14.57) - open/close 6009 (time 1709787761.81 total 343.95 last 10.56) - open/close 6205 (time 1709787771.81 total 353.95 last 19.60) - open/close 6312 (time 1709787781.98 total 364.12 last 10.52) - open/close 6479 (time 1709787792.57 total 374.71 last 15.77) - open/close 6662 (time 1709787802.82 total 384.96 last 17.85) - open/close 6803 (time 1709787813.01 total 395.15 last 13.83) - open/close 6915 (time 1709787823.45 total 405.60 last 10.73) - open/close 7089 (time 1709787833.46 total 415.60 last 17.39) - open/close 7335 (time 1709787843.48 total 425.62 last 24.56) - open/close 7444 (time 1709787853.99 total 436.14 last 10.37) - open/close 7638 (time 1709787864.86 total 447.00 last 17.86) - open/close 7783 (time 1709787874.87 total 457.01 last 14.48) - open/close 7861 (time 1709787886.85 total 468.99 last 6.51) - open/close 7974 (time 1709787897.52 total 479.66 last 10.59) - open/close 8169 (time 1709787908.20 total 490.34 last 18.25) - open/close 8367 (time 1709787918.21 total 500.35 last 19.79) - open/close 8537 (time 1709787928.22 total 510.36 last 16.98) - open/close 8709 (time 1709787938.70 total 520.84 last 16.42) - open/close 8907 (time 1709787949.22 total 531.36 last 18.82) - open/close 9085 (time 1709787959.24 total 541.38 last 17.76) - open/close 9333 (time 1709787970.14 total 552.28 last 22.75) - open/close 9593 (time 1709787980.59 total 562.73 last 24.89) - open/close 9778 (time 1709787991.59 total 573.73 last 16.82) - open/close 10000 (time 1709787997.81 total 579.95 last 35.67) - open/close 10756 (time 1709788007.82 total 589.96 last 75.55) - open/close 11510 (time 1709788017.82 total 599.97 last 75.36) - open/close 12314 (time 1709788027.83 total 609.97 last 80.40) - open/close 13101 (time 1709788037.83 total 619.97 last 78.70) - open/close 13841 (time 1709788047.83 total 629.97 last 73.99) - open/close 14585 (time 1709788057.83 total 639.97 last 74.36) - open/close 15235 (time 1709788067.83 total 649.97 last 64.99) - open/close 15858 (time 1709788077.83 total 659.97 last 62.30) - open/close 16485 (time 1709788087.84 total 669.98 last 62.67) - open/close 17077 (time 1709788097.85 total 679.99 last 59.14) - open/close 17659 (time 1709788107.85 total 689.99 last 58.18) - open/close 18207 (time 1709788117.87 total 700.01 last 54.72) - open/close 18772 (time 1709788127.87 total 710.01 last 56.48) - open/close 19326 (time 1709788137.89 total 720.03 last 55.29) - open/close 19855 (time 1709788147.89 total 730.03 last 52.89) - open/close 20000 (time 1709788150.56 total 732.70 last 54.40) - open/close 20549 (time 1709788160.56 total 742.70 last 54.89) - open/close 21086 (time 1709788170.57 total 752.71 last 53.64) - open/close 21616 (time 1709788180.58 total 762.73 last 52.92) - open/close 22109 (time 1709788190.60 total 772.74 last 49.21) - open/close 22585 (time 1709788200.61 total 782.75 last 47.55) - open/close 23072 (time 1709788210.62 total 792.76 last 48.68) - open/close 23564 (time 1709788220.62 total 802.76 last 49.20) - open/close 24037 (time 1709788230.62 total 812.77 last 47.27) - open/close 24494 (time 1709788240.64 total 822.79 last 45.61) - open/close 24943 (time 1709788250.66 total 832.80 last 44.82) - open/close 25382 (time 1709788260.66 total 842.81 last 43.89) - open/close 25828 (time 1709788270.67 total 852.81 last 44.59) - open/close 26263 (time 1709788280.68 total 862.82 last 43.44) - open/close 26648 (time 1709788290.68 total 872.82 last 38.49) - open/close 27038 (time 1709788300.69 total 882.83 last 38.98) - open/close 27418 (time 1709788310.74 total 892.88 last 37.79) - open/close 27793 (time 1709788320.75 total 902.89 last 37.46) - open/close 28158 (time 1709788330.77 total 912.91 last 36.42) - open/close 28510 (time 1709788340.78 total 922.92 last 35.18) - open/close 28869 (time 1709788350.80 total 932.94 last 35.82) - open/close 29232 (time 1709788360.81 total 942.95 last 36.28) - open/close 29592 (time 1709788370.83 total 952.97 last 35.91) - open/close 29940 (time 1709788380.85 total 962.99 last 34.73) - open/close 30276 (time 1709788390.88 total 973.02 last 33.52) - open/close 30619 (time 1709788400.89 total 983.03 last 34.26) - open/close 30996 (time 1709788410.91 total 993.05 last 37.62) - open/close 31370 (time 1709788420.91 total 1003.06 last 37.39) - open/close 31751 (time 1709788430.92 total 1013.07 last 38.06) - open/close 32137 (time 1709788440.93 total 1023.07 last 38.59) - open/close 32516 (time 1709788450.94 total 1033.08 last 37.84) - open/close 32867 (time 1709788460.97 total 1043.11 last 35.01) - open/close 33214 (time 1709788470.98 total 1053.12 last 34.65) - open/close 33583 (time 1709788480.99 total 1063.13 last 36.86) - open/close 33958 (time 1709788491.00 total 1073.14 last 37.48) - open/close 34337 (time 1709788501.00 total 1083.15 last 37.88) - open/close 34731 (time 1709788511.02 total 1093.16 last 39.34) - open/close 35106 (time 1709788521.04 total 1103.18 last 37.44) - open/close 35471 (time 1709788531.05 total 1113.20 last 36.43) - open/close 35850 (time 1709788541.08 total 1123.22 last 37.81) - open/close 36219 (time 1709788551.09 total 1133.23 last 36.87) - open/close 36578 (time 1709788561.11 total 1143.25 last 35.83) - open/close 36935 (time 1709788571.14 total 1153.28 last 35.59) - open/close 37280 (time 1709788581.16 total 1163.30 last 34.43) - open/close 37641 (time 1709788591.18 total 1173.32 last 36.02) - open/close 37991 (time 1709788601.18 total 1183.33 last 34.99) - open/close 38341 (time 1709788611.21 total 1193.35 last 34.91) - open/close 38698 (time 1709788621.24 total 1203.38 last 35.60) - open/close 39037 (time 1709788631.24 total 1213.38 last 33.89) - open/close 39360 (time 1709788641.25 total 1223.39 last 32.26) - open/close 39666 (time 1709788651.25 total 1233.40 last 30.60) - open/close 39982 (time 1709788661.26 total 1243.40 last 31.59) - open/close 40307 (time 1709788671.27 total 1253.41 last 32.47) - open/close 40621 (time 1709788681.28 total 1263.42 last 31.36) - open/close 40939 (time 1709788691.30 total 1273.44 last 31.73) - open/close 41257 (time 1709788701.30 total 1283.44 last 31.80) - open/close 41574 (time 1709788711.31 total 1293.45 last 31.69) - open/close 41907 (time 1709788721.34 total 1303.48 last 33.19) - open/close 42227 (time 1709788731.37 total 1313.51 last 31.91) - open/close 42551 (time 1709788741.39 total 1323.53 last 32.34) - open/close 42872 (time 1709788751.41 total 1333.55 last 32.02) - open/close 43192 (time 1709788761.43 total 1343.57 last 31.95) - open/close 43513 (time 1709788771.43 total 1353.57 last 32.09) - open/close 43823 (time 1709788781.45 total 1363.59 last 30.94) - open/close 44122 (time 1709788791.49 total 1373.63 last 29.79) - open/close 44433 (time 1709788801.50 total 1383.64 last 31.06) - open/close 44744 (time 1709788811.50 total 1393.65 last 31.08) - open/close 45040 (time 1709788821.52 total 1403.66 last 29.55) - open/close 45329 (time 1709788831.55 total 1413.69 last 28.82) - open/close 45623 (time 1709788841.56 total 1423.70 last 29.37) - open/close 45922 (time 1709788851.58 total 1433.72 last 29.84) - open/close 46218 (time 1709788861.58 total 1443.73 last 29.59) - open/close 46515 (time 1709788871.60 total 1453.74 last 29.66) - open/close 46802 (time 1709788881.60 total 1463.75 last 28.68) - open/close 47093 (time 1709788891.63 total 1473.77 last 29.02) - open/close 47385 (time 1709788901.65 total 1483.79 last 29.15) - open/close 47677 (time 1709788911.65 total 1493.79 last 29.19) - open/close 47959 (time 1709788921.68 total 1503.82 last 28.11) - open/close 48243 (time 1709788931.69 total 1513.84 last 28.36) - open/close 48524 (time 1709788941.72 total 1523.86 last 28.02) - open/close 48796 (time 1709788951.72 total 1533.87 last 27.19) - open/close 49059 (time 1709788961.75 total 1543.89 last 26.23) - open/close 49331 (time 1709788971.77 total 1553.92 last 27.13) - open/close 49606 (time 1709788981.78 total 1563.92 last 27.49) - open/close 49876 (time 1709788991.79 total 1573.94 last 26.96) total: 50000 open/close in 1578.62 seconds: 31.67 ops/second - unlinked 0 (time 1709788997 ; total 0 ; last 0) - unlinked 10000 (time 1709789433 ; total 436 ; last 436) - unlinked 20000 (time 1709789788 ; total 791 ; last 355) - unlinked 30000 (time 1709790024 ; total 1027 ; last 236) - unlinked 40000 (time 1709790124 ; total 1127 ; last 100) total: 50000 unlinks in 1155 seconds: 43.290043 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server unloading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-client: ssh exited with exit code 2 pdsh@oleg205-client: oleg205-server: ssh exited with exit code 2