== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:52:15 (1786755135) start mds service on oleg307-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg307-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1786755242.91 total 11.04 last 17.75) - open/close 489 (time 1786755252.91 total 21.04 last 29.29) - open/close 724 (time 1786755263.05 total 31.18 last 23.17) - open/close 937 (time 1786755273.08 total 41.21 last 21.24) - open/close 1121 (time 1786755283.09 total 51.23 last 18.37) - open/close 1345 (time 1786755293.10 total 61.24 last 22.38) - open/close 1575 (time 1786755303.43 total 71.57 last 22.27) - open/close 1797 (time 1786755314.52 total 82.66 last 20.01) - open/close 2044 (time 1786755324.54 total 92.68 last 24.65) - open/close 2299 (time 1786755334.57 total 102.70 last 25.43) - open/close 2541 (time 1786755345.24 total 113.37 last 22.69) - open/close 2713 (time 1786755355.53 total 123.67 last 16.71) - open/close 2996 (time 1786755366.48 total 134.62 last 25.84) - open/close 3277 (time 1786755382.56 total 150.70 last 17.48) - open/close 3542 (time 1786755395.54 total 163.67 last 20.42) - open/close 3776 (time 1786755406.03 total 174.17 last 22.30) - open/close 4053 (time 1786755417.04 total 185.18 last 25.15) - open/close 4260 (time 1786755427.08 total 195.22 last 20.62) - open/close 4447 (time 1786755437.77 total 205.91 last 17.50) - open/close 4572 (time 1786755447.80 total 215.94 last 12.46) - open/close 4725 (time 1786755458.32 total 226.46 last 14.54) - open/close 4914 (time 1786755468.33 total 236.47 last 18.88) - open/close 5101 (time 1786755480.82 total 248.95 last 14.98) - open/close 5290 (time 1786755490.83 total 258.97 last 18.87) - open/close 5462 (time 1786755501.09 total 269.23 last 16.76) - open/close 5749 (time 1786755512.17 total 280.30 last 25.92) - open/close 5976 (time 1786755522.17 total 290.31 last 22.68) - open/close 6230 (time 1786755532.19 total 300.33 last 25.36) - open/close 6484 (time 1786755542.20 total 310.34 last 25.36) - open/close 6705 (time 1786755552.47 total 320.61 last 21.53) - open/close 6909 (time 1786755562.72 total 330.85 last 19.91) - open/close 7037 (time 1786755573.59 total 341.73 last 11.77) - open/close 7160 (time 1786755584.36 total 352.49 last 11.42) - open/close 7404 (time 1786755594.36 total 362.50 last 24.38) - open/close 7693 (time 1786755605.33 total 373.46 last 26.36) - open/close 7973 (time 1786755615.33 total 383.47 last 27.98) - open/close 8109 (time 1786755626.39 total 394.52 last 12.30) - open/close 8336 (time 1786755636.57 total 404.71 last 22.29) - open/close 8561 (time 1786755646.94 total 415.08 last 21.69) - open/close 8840 (time 1786755657.63 total 425.76 last 26.11) - open/close 9048 (time 1786755667.64 total 435.77 last 20.79) - open/close 9307 (time 1786755678.17 total 446.30 last 24.59) - open/close 9567 (time 1786755688.53 total 456.67 last 25.08) - open/close 9786 (time 1786755699.15 total 467.29 last 20.62) - open/close 10000 (time 1786755708.03 total 476.17 last 24.11) - open/close 10209 (time 1786755719.32 total 487.45 last 18.52) - open/close 10290 (time 1786755729.32 total 497.46 last 8.10) - open/close 10547 (time 1786755739.63 total 507.76 last 24.93) - open/close 10794 (time 1786755750.11 total 518.25 last 23.56) - open/close 10998 (time 1786755760.11 total 528.25 last 20.40) - open/close 11089 (time 1786755770.26 total 538.40 last 8.96) - open/close 11242 (time 1786755780.27 total 548.41 last 15.29) - open/close 11428 (time 1786755791.19 total 559.32 last 17.04) - open/close 11550 (time 1786755801.20 total 569.33 last 12.18) - open/close 11756 (time 1786755811.21 total 579.34 last 20.58) - open/close 11962 (time 1786755821.21 total 589.34 last 20.59) - open/close 12134 (time 1786755831.59 total 599.73 last 16.56) - open/close 12389 (time 1786755841.61 total 609.74 last 25.47) - open/close 12629 (time 1786755851.61 total 619.75 last 23.99) - open/close 12890 (time 1786755861.91 total 630.05 last 25.34) - open/close 13113 (time 1786755871.92 total 640.06 last 22.27) - open/close 13375 (time 1786755881.92 total 650.06 last 26.20) - open/close 13580 (time 1786755891.94 total 660.07 last 20.47) - open/close 13796 (time 1786755902.18 total 670.31 last 21.09) - open/close 14069 (time 1786755912.20 total 680.34 last 27.23) - open/close 14302 (time 1786755923.22 total 691.36 last 21.14) - open/close 14499 (time 1786755937.47 total 705.60 last 13.83) - open/close 14755 (time 1786755948.18 total 716.31 last 23.90) - open/close 14992 (time 1786755959.19 total 727.33 last 21.51) - open/close 15271 (time 1786755969.20 total 737.34 last 27.88) - open/close 15507 (time 1786755979.66 total 747.80 last 22.56) - open/close 15778 (time 1786755990.13 total 758.27 last 25.89) - open/close 16060 (time 1786756000.14 total 768.27 last 28.19) - open/close 16298 (time 1786756010.51 total 778.65 last 22.94) - open/close 16546 (time 1786756020.52 total 788.65 last 24.79) - open/close 16822 (time 1786756030.55 total 798.69 last 27.51) - open/close 16972 (time 1786756041.63 total 809.76 last 13.54) - open/close 17183 (time 1786756054.04 total 822.17 last 17.00) - open/close 17357 (time 1786756064.10 total 832.23 last 17.30) - open/close 17531 (time 1786756074.31 total 842.45 last 17.04) - open/close 17764 (time 1786756084.33 total 852.46 last 23.27) - open/close 18049 (time 1786756094.33 total 862.47 last 28.49) - open/close 18271 (time 1786756104.53 total 872.67 last 21.76) - open/close 18527 (time 1786756115.03 total 883.16 last 24.40) - open/close 18781 (time 1786756125.78 total 893.92 last 23.61) - open/close 18966 (time 1786756135.80 total 903.93 last 18.48) - open/close 19214 (time 1786756146.40 total 914.53 last 23.39) - open/close 19503 (time 1786756157.46 total 925.59 last 26.14) - open/close 19757 (time 1786756167.46 total 935.59 last 25.40) - open/close 19992 (time 1786756177.46 total 945.60 last 23.49) - open/close 20457 (time 1786756187.47 total 955.61 last 46.46) - open/close 21120 (time 1786756197.47 total 965.61 last 66.30) - open/close 21893 (time 1786756207.48 total 975.61 last 77.25) - open/close 22650 (time 1786756217.48 total 985.62 last 75.65) - open/close 23451 (time 1786756227.49 total 995.62 last 80.09) - open/close 24205 (time 1786756237.49 total 1005.62 last 75.40) - open/close 25027 (time 1786756247.49 total 1015.62 last 82.20) - open/close 25781 (time 1786756257.49 total 1025.63 last 75.35) - open/close 26575 (time 1786756267.50 total 1035.64 last 79.35) - open/close 27368 (time 1786756277.51 total 1045.65 last 79.21) - open/close 28100 (time 1786756287.52 total 1055.65 last 73.14) - open/close 28826 (time 1786756297.53 total 1065.66 last 72.55) - open/close 29509 (time 1786756307.54 total 1075.67 last 68.23) - open/close 30000 (time 1786756314.53 total 1082.66 last 70.24) - open/close 30729 (time 1786756324.53 total 1092.67 last 72.85) - open/close 31506 (time 1786756334.55 total 1102.68 last 77.57) - open/close 32251 (time 1786756344.56 total 1112.69 last 74.46) - open/close 33003 (time 1786756354.56 total 1122.70 last 75.15) - open/close 33759 (time 1786756364.57 total 1132.71 last 75.52) - open/close 34527 (time 1786756374.57 total 1142.71 last 76.80) - open/close 35286 (time 1786756384.58 total 1152.72 last 75.83) - open/close 36069 (time 1786756394.59 total 1162.72 last 78.28) - open/close 36874 (time 1786756404.59 total 1172.73 last 80.45) - open/close 37687 (time 1786756414.60 total 1182.73 last 81.27) - open/close 38469 (time 1786756424.60 total 1192.74 last 78.16) - open/close 39263 (time 1786756434.62 total 1202.75 last 79.27) - open/close 40000 (time 1786756443.80 total 1211.94 last 80.23) - open/close 40781 (time 1786756453.80 total 1221.94 last 78.09) - open/close 41567 (time 1786756463.83 total 1231.96 last 78.43) - open/close 42269 (time 1786756473.83 total 1241.97 last 70.17) - open/close 43000 (time 1786756483.84 total 1251.97 last 73.04) - open/close 43678 (time 1786756493.84 total 1261.97 last 67.79) - open/close 44502 (time 1786756503.85 total 1271.98 last 82.35) - open/close 45173 (time 1786756513.86 total 1281.99 last 67.04) - open/close 45995 (time 1786756523.86 total 1292.00 last 82.13) - open/close 46812 (time 1786756533.87 total 1302.00 last 81.66) - open/close 47591 (time 1786756543.87 total 1312.01 last 77.88) - open/close 48456 (time 1786756553.87 total 1322.01 last 86.48) - open/close 49220 (time 1786756563.88 total 1332.01 last 76.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1337.14 seconds: 37.12 ops/second - unlinked 0 (time 1786756573 ; total 0 ; last 0) - unlinked 10000 (time 1786756712 ; total 139 ; last 139) - unlinked 20000 (time 1786756845 ; total 272 ; last 133) - unlinked 30000 (time 1786756982 ; total 409 ; last 137) - unlinked 40000 (time 1786757122 ; total 549 ; last 140) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 688 seconds: 72.138084 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2