== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:37 (1709787157) start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock oleg255-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 200 (time 1709787202.44 total 10.00 last 19.99) - open/close 445 (time 1709787212.95 total 20.52 last 23.31) - open/close 592 (time 1709787223.97 total 31.54 last 13.34) - open/close 777 (time 1709787234.74 total 42.31 last 17.17) - open/close 999 (time 1709787245.15 total 52.72 last 21.34) - open/close 1261 (time 1709787255.16 total 62.73 last 26.17) - open/close 1466 (time 1709787265.33 total 72.90 last 20.16) - open/close 1645 (time 1709787276.26 total 83.83 last 16.37) - open/close 1748 (time 1709787286.86 total 94.43 last 9.72) - open/close 1937 (time 1709787297.00 total 104.57 last 18.64) - open/close 2188 (time 1709787307.01 total 114.58 last 25.08) - open/close 2446 (time 1709787317.54 total 125.11 last 24.51) - open/close 2533 (time 1709787327.98 total 135.55 last 8.33) - open/close 2653 (time 1709787338.00 total 145.57 last 11.97) - open/close 2846 (time 1709787348.83 total 156.40 last 17.83) - open/close 3034 (time 1709787359.44 total 167.01 last 17.72) - open/close 3206 (time 1709787370.13 total 177.70 last 16.09) - open/close 3412 (time 1709787380.98 total 188.55 last 18.98) - open/close 3576 (time 1709787391.00 total 198.57 last 16.37) - open/close 3811 (time 1709787401.00 total 208.57 last 23.50) - open/close 3925 (time 1709787411.49 total 219.06 last 10.87) - open/close 4050 (time 1709787421.71 total 229.28 last 12.24) - open/close 4278 (time 1709787431.73 total 239.30 last 22.75) - open/close 4365 (time 1709787449.71 total 257.28 last 4.84) - open/close 4563 (time 1709787460.52 total 268.09 last 18.31) - open/close 4687 (time 1709787470.97 total 278.54 last 11.87) - open/close 4845 (time 1709787480.98 total 288.55 last 15.78) - open/close 5020 (time 1709787491.04 total 298.61 last 17.40) - open/close 5142 (time 1709787501.31 total 308.88 last 11.88) - open/close 5314 (time 1709787511.86 total 319.43 last 16.30) - open/close 5500 (time 1709787521.96 total 329.53 last 18.42) - open/close 5637 (time 1709787531.98 total 339.55 last 13.67) - open/close 5804 (time 1709787541.99 total 349.56 last 16.69) - open/close 6040 (time 1709787552.92 total 360.49 last 21.59) - open/close 6174 (time 1709787562.94 total 370.51 last 13.37) - open/close 6415 (time 1709787572.94 total 380.51 last 24.09) - open/close 6616 (time 1709787582.96 total 390.52 last 20.08) - open/close 6763 (time 1709787593.90 total 401.47 last 13.43) - open/close 6911 (time 1709787605.01 total 412.58 last 13.32) - open/close 7118 (time 1709787615.68 total 423.25 last 19.40) - open/close 7262 (time 1709787625.70 total 433.27 last 14.37) - open/close 7535 (time 1709787636.68 total 444.24 last 24.88) - open/close 7748 (time 1709787647.70 total 455.27 last 19.32) - open/close 7856 (time 1709787658.02 total 465.59 last 10.46) - open/close 8059 (time 1709787668.07 total 475.64 last 20.19) - open/close 8274 (time 1709787678.67 total 486.24 last 20.28) - open/close 8465 (time 1709787689.04 total 496.60 last 18.43) - open/close 8607 (time 1709787699.73 total 507.30 last 13.28) - open/close 8736 (time 1709787709.76 total 517.33 last 12.86) - open/close 8856 (time 1709787719.79 total 527.36 last 11.97) - open/close 8999 (time 1709787730.78 total 538.35 last 13.01) - open/close 9108 (time 1709787740.79 total 548.35 last 10.90) - open/close 9223 (time 1709787750.81 total 558.38 last 11.48) - open/close 9453 (time 1709787760.96 total 568.53 last 22.65) - open/close 9584 (time 1709787770.96 total 578.53 last 13.10) - open/close 9760 (time 1709787780.99 total 588.56 last 17.55) - open/close 10000 (time 1709787790.23 total 597.80 last 25.96) - open/close 10596 (time 1709787800.24 total 607.80 last 59.59) - open/close 11148 (time 1709787810.26 total 617.83 last 55.08) - open/close 11646 (time 1709787820.27 total 627.84 last 49.74) - open/close 11826 (time 1709787835.16 total 642.72 last 12.09) - open/close 12313 (time 1709787845.16 total 652.73 last 48.66) - open/close 12759 (time 1709787855.16 total 662.73 last 44.60) - open/close 13198 (time 1709787865.17 total 672.74 last 43.87) - open/close 13714 (time 1709787875.18 total 682.75 last 51.53) - open/close 14184 (time 1709787885.19 total 692.76 last 46.97) - open/close 14600 (time 1709787895.20 total 702.77 last 41.57) - open/close 15005 (time 1709787905.21 total 712.78 last 40.47) - open/close 15417 (time 1709787915.21 total 722.78 last 41.18) - open/close 15855 (time 1709787925.21 total 732.78 last 43.79) - open/close 16284 (time 1709787935.29 total 742.86 last 42.56) - open/close 16723 (time 1709787945.31 total 752.88 last 43.84) - open/close 17218 (time 1709787955.32 total 762.89 last 49.44) - open/close 17715 (time 1709787965.34 total 772.91 last 49.60) - open/close 18156 (time 1709787975.34 total 782.91 last 44.10) - open/close 18592 (time 1709787985.36 total 792.93 last 43.51) - open/close 19041 (time 1709787995.37 total 802.94 last 44.86) - open/close 19552 (time 1709788005.38 total 812.95 last 51.05) - open/close 20000 (time 1709788014.04 total 821.60 last 51.74) - open/close 20544 (time 1709788024.05 total 831.62 last 54.34) - open/close 21080 (time 1709788034.07 total 841.63 last 53.50) - open/close 21621 (time 1709788044.07 total 851.64 last 54.07) - open/close 22154 (time 1709788054.08 total 861.65 last 53.25) - open/close 22654 (time 1709788064.09 total 871.65 last 49.98) - open/close 23128 (time 1709788074.09 total 881.66 last 47.38) - open/close 23667 (time 1709788084.11 total 891.67 last 53.81) - open/close 24152 (time 1709788094.11 total 901.68 last 48.48) - open/close 24627 (time 1709788104.11 total 911.68 last 47.48) - open/close 25110 (time 1709788114.13 total 921.69 last 48.24) - open/close 25566 (time 1709788124.15 total 931.72 last 45.51) - open/close 26039 (time 1709788134.15 total 941.72 last 47.29) - open/close 26514 (time 1709788144.16 total 951.73 last 47.45) - open/close 26977 (time 1709788154.18 total 961.75 last 46.21) - open/close 27455 (time 1709788164.19 total 971.75 last 47.76) - open/close 27926 (time 1709788174.19 total 981.76 last 47.07) - open/close 28369 (time 1709788184.20 total 991.77 last 44.27) - open/close 28838 (time 1709788194.21 total 1001.78 last 46.85) - open/close 29276 (time 1709788204.21 total 1011.78 last 43.79) - open/close 29732 (time 1709788214.22 total 1021.78 last 45.59) - open/close 30000 (time 1709788220.09 total 1027.66 last 45.63) - open/close 30437 (time 1709788230.11 total 1037.68 last 43.60) - open/close 30856 (time 1709788240.11 total 1047.68 last 41.90) - open/close 31289 (time 1709788250.13 total 1057.70 last 43.23) - open/close 31722 (time 1709788260.14 total 1067.71 last 43.23) - open/close 32109 (time 1709788270.15 total 1077.72 last 38.69) - open/close 32523 (time 1709788280.15 total 1087.72 last 41.39) - open/close 32893 (time 1709788290.16 total 1097.73 last 36.95) - open/close 33288 (time 1709788300.17 total 1107.74 last 39.49) - open/close 33676 (time 1709788310.17 total 1117.74 last 38.77) - open/close 34060 (time 1709788320.19 total 1127.76 last 38.33) - open/close 34435 (time 1709788330.21 total 1137.78 last 37.44) - open/close 34808 (time 1709788340.22 total 1147.79 last 37.24) - open/close 35173 (time 1709788350.23 total 1157.80 last 36.46) - open/close 35533 (time 1709788360.24 total 1167.81 last 35.97) - open/close 35893 (time 1709788370.25 total 1177.81 last 35.99) - open/close 36241 (time 1709788380.25 total 1187.82 last 34.78) - open/close 36574 (time 1709788390.28 total 1197.85 last 33.22) - open/close 36933 (time 1709788400.29 total 1207.86 last 35.85) - open/close 37291 (time 1709788410.29 total 1217.86 last 35.79) - open/close 37647 (time 1709788420.32 total 1227.89 last 35.49) - open/close 37993 (time 1709788430.34 total 1237.91 last 34.55) - open/close 38330 (time 1709788440.35 total 1247.91 last 33.67) - open/close 38669 (time 1709788450.35 total 1257.91 last 33.90) - open/close 39011 (time 1709788460.37 total 1267.94 last 34.11) - open/close 39342 (time 1709788470.39 total 1277.95 last 33.06) - open/close 39702 (time 1709788480.39 total 1287.96 last 35.97) - open/close 40000 (time 1709788489.50 total 1297.06 last 32.74) - open/close 40336 (time 1709788499.51 total 1307.08 last 33.54) - open/close 40679 (time 1709788509.52 total 1317.09 last 34.27) - open/close 41021 (time 1709788519.53 total 1327.10 last 34.17) - open/close 41352 (time 1709788529.56 total 1337.12 last 33.02) - open/close 41685 (time 1709788539.57 total 1347.14 last 33.26) - open/close 42008 (time 1709788549.58 total 1357.14 last 32.28) - open/close 42341 (time 1709788559.59 total 1367.15 last 33.26) - open/close 42676 (time 1709788569.61 total 1377.17 last 33.43) - open/close 43006 (time 1709788579.63 total 1387.20 last 32.91) - open/close 43341 (time 1709788589.64 total 1397.21 last 33.47) - open/close 43682 (time 1709788599.66 total 1407.22 last 34.05) - open/close 44010 (time 1709788609.69 total 1417.25 last 32.70) - open/close 44343 (time 1709788619.71 total 1427.28 last 33.22) - open/close 44662 (time 1709788629.71 total 1437.28 last 31.89) - open/close 44965 (time 1709788639.73 total 1447.30 last 30.24) - open/close 45269 (time 1709788649.76 total 1457.32 last 30.33) - open/close 45569 (time 1709788659.78 total 1467.35 last 29.92) - open/close 45872 (time 1709788669.80 total 1477.36 last 30.26) - open/close 46179 (time 1709788679.82 total 1487.39 last 30.64) - open/close 46472 (time 1709788689.83 total 1497.40 last 29.25) - open/close 46770 (time 1709788699.83 total 1507.40 last 29.80) - open/close 47061 (time 1709788709.86 total 1517.43 last 29.02) - open/close 47365 (time 1709788719.89 total 1527.46 last 30.32) - open/close 47662 (time 1709788729.89 total 1537.46 last 29.69) - open/close 47961 (time 1709788739.90 total 1547.47 last 29.86) - open/close 48246 (time 1709788749.94 total 1557.51 last 28.40) - open/close 48524 (time 1709788759.97 total 1567.54 last 27.72) - open/close 48810 (time 1709788769.97 total 1577.54 last 28.58) - open/close 49086 (time 1709788779.98 total 1587.54 last 27.60) - open/close 49362 (time 1709788790.01 total 1597.58 last 27.51) - open/close 49652 (time 1709788800.02 total 1607.58 last 28.98) - open/close 49928 (time 1709788810.04 total 1617.61 last 27.53) total: 50000 open/close in 1620.19 seconds: 30.86 ops/second - unlinked 0 (time 1709788813 ; total 0 ; last 0) - unlinked 10000 (time 1709789270 ; total 457 ; last 457) - unlinked 20000 (time 1709789659 ; total 846 ; last 389) - unlinked 30000 (time 1709789925 ; total 1112 ; last 266) - unlinked 40000 (time 1709790039 ; total 1226 ; last 114) total: 50000 unlinks in 1267 seconds: 39.463299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2