== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:17:01 (1709785021) start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 683 (time 1709785047.50 total 10.37 last 65.83) - open/close 1528 (time 1709785058.27 total 21.14 last 78.49) - open/close 2078 (time 1709785068.65 total 31.53 last 52.96) - open/close 2430 (time 1709785078.65 total 41.53 last 35.20) - open/close 2926 (time 1709785089.07 total 51.94 last 47.62) - open/close 3310 (time 1709785099.07 total 61.94 last 38.39) - open/close 3851 (time 1709785109.40 total 72.28 last 52.36) - open/close 4424 (time 1709785119.90 total 82.77 last 54.61) - open/close 5039 (time 1709785130.56 total 93.43 last 57.68) - open/close 5640 (time 1709785141.39 total 104.26 last 55.50) - open/close 6180 (time 1709785152.23 total 115.11 last 49.78) - open/close 6603 (time 1709785162.76 total 125.63 last 40.19) - open/close 6929 (time 1709785173.05 total 135.92 last 31.67) - open/close 7409 (time 1709785183.84 total 146.71 last 44.50) - open/close 7679 (time 1709785194.17 total 157.04 last 26.14) - open/close 8023 (time 1709785204.38 total 167.25 last 33.68) - open/close 8539 (time 1709785214.39 total 177.26 last 51.57) - open/close 8966 (time 1709785225.19 total 188.06 last 39.52) - open/close 9396 (time 1709785235.19 total 198.06 last 42.99) - open/close 9795 (time 1709785245.94 total 208.81 last 37.11) - open/close 10000 (time 1709785250.84 total 213.71 last 41.84) - open/close 10223 (time 1709785260.91 total 223.78 last 22.15) - open/close 10612 (time 1709785270.91 total 233.79 last 38.89) - open/close 11423 (time 1709785280.92 total 243.79 last 81.07) - open/close 12915 (time 1709785290.92 total 253.79 last 149.19) - open/close 14262 (time 1709785300.92 total 263.80 last 134.60) - open/close 15706 (time 1709785310.92 total 273.80 last 144.39) - open/close 16917 (time 1709785320.93 total 283.80 last 121.02) - open/close 18308 (time 1709785330.94 total 293.81 last 139.03) - open/close 19472 (time 1709785340.94 total 303.81 last 116.39) - open/close 20000 (time 1709785345.45 total 308.32 last 117.11) - open/close 21117 (time 1709785355.45 total 318.32 last 111.64) - open/close 22160 (time 1709785365.46 total 328.33 last 104.24) - open/close 23095 (time 1709785375.46 total 338.33 last 93.50) - open/close 23959 (time 1709785385.46 total 348.34 last 86.34) - open/close 24751 (time 1709785395.47 total 358.35 last 79.11) - open/close 25524 (time 1709785405.48 total 368.36 last 77.23) - open/close 26204 (time 1709785415.48 total 378.36 last 67.99) - open/close 27048 (time 1709785425.49 total 388.37 last 84.31) - open/close 27770 (time 1709785435.50 total 398.38 last 72.15) - open/close 28501 (time 1709785445.52 total 408.39 last 72.99) - open/close 29235 (time 1709785455.52 total 418.39 last 73.38) - open/close 29971 (time 1709785465.53 total 428.40 last 73.56) - open/close 30642 (time 1709785475.53 total 438.41 last 67.04) - open/close 31211 (time 1709785485.55 total 448.42 last 56.82) - open/close 31865 (time 1709785495.55 total 458.43 last 65.37) - open/close 32473 (time 1709785505.57 total 468.45 last 60.68) - open/close 33127 (time 1709785515.58 total 478.46 last 65.33) - open/close 33654 (time 1709785525.59 total 488.46 last 52.69) - open/close 34225 (time 1709785535.60 total 498.47 last 57.02) - open/close 34795 (time 1709785545.60 total 508.48 last 56.98) - open/close 35383 (time 1709785555.60 total 518.48 last 58.80) - open/close 35954 (time 1709785565.61 total 528.49 last 57.05) - open/close 36548 (time 1709785575.63 total 538.50 last 59.31) - open/close 37120 (time 1709785585.65 total 548.52 last 57.09) - open/close 37625 (time 1709785595.65 total 558.53 last 50.46) - open/close 38174 (time 1709785605.66 total 568.54 last 54.86) - open/close 38702 (time 1709785615.67 total 578.55 last 52.75) - open/close 39157 (time 1709785625.68 total 588.55 last 45.47) - open/close 39645 (time 1709785635.68 total 598.55 last 48.80) - open/close 40000 (time 1709785642.51 total 605.38 last 51.98) - open/close 40457 (time 1709785652.52 total 615.40 last 45.63) - open/close 40909 (time 1709785662.54 total 625.41 last 45.13) - open/close 41412 (time 1709785672.55 total 635.42 last 50.25) - open/close 41880 (time 1709785682.55 total 645.43 last 46.79) - open/close 42330 (time 1709785692.56 total 655.43 last 44.97) - open/close 42779 (time 1709785702.59 total 665.46 last 44.77) - open/close 43187 (time 1709785712.59 total 675.47 last 40.77) - open/close 43601 (time 1709785722.60 total 685.47 last 41.39) - open/close 44086 (time 1709785732.60 total 695.47 last 48.48) - open/close 44534 (time 1709785742.61 total 705.49 last 44.74) - open/close 44995 (time 1709785752.63 total 715.51 last 46.02) - open/close 45429 (time 1709785762.63 total 725.51 last 43.40) - open/close 45875 (time 1709785772.65 total 735.52 last 44.54) - open/close 46294 (time 1709785782.65 total 745.52 last 41.90) - open/close 46729 (time 1709785792.65 total 755.52 last 43.48) - open/close 47166 (time 1709785802.67 total 765.55 last 43.60) - open/close 47550 (time 1709785812.68 total 775.55 last 38.39) - open/close 48013 (time 1709785822.69 total 785.56 last 46.26) - open/close 48423 (time 1709785832.69 total 795.56 last 40.98) - open/close 48851 (time 1709785842.69 total 805.56 last 42.80) - open/close 49221 (time 1709785852.72 total 815.59 last 36.90) - open/close 49592 (time 1709785862.74 total 825.61 last 37.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 828.37 seconds: 59.99 ops/second - unlinked 0 (time 1709785866 ; total 0 ; last 0) - unlinked 10000 (time 1709786147 ; total 281 ; last 281) - unlinked 20000 (time 1709786354 ; total 488 ; last 207) - unlinked 30000 (time 1709786478 ; total 612 ; last 124) - unlinked 40000 (time 1709786540 ; total 674 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 713 seconds: 69.702667 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2