== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:18:59 (1709785139) start mds service on oleg358-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg358-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock oleg358-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 815 (time 1709785169.80 total 10.76 last 75.74) - open/close 1653 (time 1709785179.80 total 20.76 last 83.79) - open/close 2126 (time 1709785189.93 total 30.90 last 46.67) - open/close 2789 (time 1709785200.56 total 41.52 last 62.39) - open/close 3489 (time 1709785211.32 total 52.29 last 65.05) - open/close 4124 (time 1709785222.05 total 63.01 last 59.21) - open/close 4739 (time 1709785232.87 total 73.83 last 56.83) - open/close 5327 (time 1709785245.68 total 86.64 last 45.91) - open/close 5920 (time 1709785255.68 total 96.64 last 59.29) - open/close 6480 (time 1709785265.68 total 106.64 last 55.99) - open/close 6768 (time 1709785275.87 total 116.83 last 28.26) - open/close 7128 (time 1709785286.60 total 127.57 last 33.54) - open/close 7460 (time 1709785297.06 total 138.02 last 31.75) - open/close 7847 (time 1709785307.72 total 148.68 last 36.30) - open/close 8267 (time 1709785318.41 total 159.38 last 39.28) - open/close 8561 (time 1709785328.70 total 169.66 last 28.58) - open/close 9022 (time 1709785338.75 total 179.72 last 45.85) - open/close 9382 (time 1709785349.37 total 190.33 last 33.92) - open/close 9780 (time 1709785359.37 total 200.33 last 39.78) - open/close 10000 (time 1709785364.39 total 205.35 last 43.87) - open/close 10500 (time 1709785375.16 total 216.13 last 46.40) - open/close 10810 (time 1709785385.83 total 226.80 last 29.05) - open/close 11140 (time 1709785395.84 total 236.80 last 32.98) - open/close 11733 (time 1709785405.84 total 246.81 last 59.26) - open/close 13127 (time 1709785415.85 total 256.81 last 139.36) - open/close 14418 (time 1709785425.85 total 266.81 last 129.10) - open/close 15639 (time 1709785435.85 total 276.81 last 122.08) - open/close 16855 (time 1709785445.85 total 286.82 last 121.54) - open/close 17979 (time 1709785455.86 total 296.82 last 112.33) - open/close 19057 (time 1709785465.86 total 306.83 last 107.77) - open/close 20000 (time 1709785474.72 total 315.68 last 106.48) - open/close 20960 (time 1709785484.73 total 325.69 last 95.90) - open/close 21899 (time 1709785494.73 total 335.70 last 93.87) - open/close 22740 (time 1709785504.74 total 345.70 last 84.08) - open/close 23572 (time 1709785514.74 total 355.71 last 83.15) - open/close 24433 (time 1709785524.75 total 365.71 last 86.03) - open/close 25268 (time 1709785534.76 total 375.72 last 83.42) - open/close 26004 (time 1709785544.77 total 385.74 last 73.51) - open/close 26693 (time 1709785554.77 total 395.74 last 68.89) - open/close 27377 (time 1709785564.79 total 405.75 last 68.33) - open/close 28110 (time 1709785574.80 total 415.76 last 73.23) - open/close 28796 (time 1709785584.80 total 425.76 last 68.59) - open/close 29438 (time 1709785594.81 total 435.77 last 64.12) - open/close 30000 (time 1709785604.03 total 444.99 last 60.98) - open/close 30578 (time 1709785614.03 total 454.99 last 57.80) - open/close 31144 (time 1709785624.03 total 464.99 last 56.59) - open/close 31725 (time 1709785634.04 total 475.00 last 58.02) - open/close 32277 (time 1709785644.06 total 485.02 last 55.10) - open/close 32822 (time 1709785654.07 total 495.03 last 54.46) - open/close 33359 (time 1709785664.07 total 505.03 last 53.69) - open/close 33924 (time 1709785674.08 total 515.05 last 56.42) - open/close 34449 (time 1709785684.10 total 525.07 last 52.40) - open/close 34976 (time 1709785694.11 total 535.08 last 52.65) - open/close 35524 (time 1709785704.13 total 545.09 last 54.73) - open/close 36032 (time 1709785714.14 total 555.10 last 50.74) - open/close 36610 (time 1709785724.15 total 565.11 last 57.72) - open/close 37157 (time 1709785734.15 total 575.11 last 54.70) - open/close 37662 (time 1709785744.16 total 585.13 last 50.43) - open/close 38083 (time 1709785754.18 total 595.14 last 42.03) - open/close 38508 (time 1709785764.19 total 605.15 last 42.47) - open/close 39009 (time 1709785774.20 total 615.16 last 50.04) - open/close 39517 (time 1709785784.21 total 625.17 last 50.76) - open/close 39980 (time 1709785794.22 total 635.18 last 46.25) - open/close 40448 (time 1709785804.23 total 645.19 last 46.75) - open/close 40894 (time 1709785814.25 total 655.21 last 44.51) - open/close 41327 (time 1709785824.26 total 665.22 last 43.27) - open/close 41784 (time 1709785834.27 total 675.23 last 45.64) - open/close 42211 (time 1709785844.28 total 685.24 last 42.66) - open/close 42659 (time 1709785854.29 total 695.26 last 44.73) - open/close 43095 (time 1709785864.30 total 705.27 last 43.56) - open/close 43543 (time 1709785874.30 total 715.27 last 44.79) - open/close 44003 (time 1709785884.31 total 725.27 last 45.99) - open/close 44445 (time 1709785894.32 total 735.28 last 44.13) - open/close 44862 (time 1709785904.34 total 745.30 last 41.62) - open/close 45284 (time 1709785914.36 total 755.32 last 42.14) - open/close 45695 (time 1709785924.37 total 765.33 last 41.05) - open/close 46096 (time 1709785934.37 total 775.33 last 40.09) - open/close 46481 (time 1709785944.38 total 785.34 last 38.46) - open/close 46879 (time 1709785954.40 total 795.36 last 39.72) - open/close 47266 (time 1709785964.41 total 805.37 last 38.66) - open/close 47678 (time 1709785974.43 total 815.40 last 41.11) - open/close 48059 (time 1709785984.44 total 825.40 last 38.09) - open/close 48439 (time 1709785994.45 total 835.42 last 37.94) - open/close 48829 (time 1709786004.48 total 845.44 last 38.90) - open/close 49210 (time 1709786014.50 total 855.46 last 38.01) - open/close 49571 (time 1709786024.51 total 865.47 last 36.07) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 866.18 seconds: 57.26 ops/second - unlinked 0 (time 1709786026 ; total 0 ; last 0) - unlinked 10000 (time 1709786321 ; total 295 ; last 295) - unlinked 20000 (time 1709786575 ; total 549 ; last 254) - unlinked 30000 (time 1709786785 ; total 759 ; last 210) - unlinked 40000 (time 1709786889 ; total 863 ; last 104) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 936 seconds: 52.991451 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2