== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:48:20 (1709056100) start mds service on oleg453-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg453-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg453-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock oleg453-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1709056152.40 total 10.80 last 19.99) - open/close 531 (time 1709056163.14 total 21.54 last 29.35) - open/close 962 (time 1709056174.08 total 32.48 last 39.39) - open/close 1305 (time 1709056184.09 total 42.49 last 34.28) - open/close 1528 (time 1709056194.90 total 53.30 last 20.63) - open/close 1778 (time 1709056204.91 total 63.31 last 24.97) - open/close 2007 (time 1709056215.00 total 73.39 last 22.70) - open/close 2247 (time 1709056225.50 total 83.90 last 22.85) - open/close 2472 (time 1709056235.51 total 93.91 last 22.48) - open/close 2785 (time 1709056245.73 total 104.13 last 30.63) - open/close 3056 (time 1709056256.68 total 115.08 last 24.73) - open/close 3322 (time 1709056267.54 total 125.94 last 24.50) - open/close 3570 (time 1709056278.33 total 136.73 last 22.98) - open/close 3849 (time 1709056288.34 total 146.74 last 27.87) - open/close 4067 (time 1709056299.02 total 157.42 last 20.42) - open/close 4266 (time 1709056309.17 total 167.57 last 19.60) - open/close 4524 (time 1709056319.43 total 177.83 last 25.16) - open/close 4810 (time 1709056329.43 total 187.83 last 28.60) - open/close 5172 (time 1709056339.69 total 198.09 last 35.27) - open/close 5327 (time 1709056349.70 total 208.10 last 15.49) - open/close 5591 (time 1709056359.72 total 218.12 last 26.33) - open/close 5894 (time 1709056369.73 total 228.13 last 30.27) - open/close 6134 (time 1709056379.74 total 238.14 last 23.99) - open/close 6321 (time 1709056389.83 total 248.23 last 18.53) - open/close 6471 (time 1709056400.23 total 258.63 last 14.43) - open/close 6766 (time 1709056410.24 total 268.64 last 29.46) - open/close 7013 (time 1709056420.25 total 278.65 last 24.69) - open/close 7300 (time 1709056430.25 total 288.65 last 28.70) - open/close 7504 (time 1709056442.01 total 300.41 last 17.35) - open/close 7649 (time 1709056452.05 total 310.45 last 14.44) - open/close 7865 (time 1709056462.55 total 320.95 last 20.57) - open/close 8265 (time 1709056473.08 total 331.48 last 37.98) - open/close 8574 (time 1709056483.73 total 342.13 last 29.01) - open/close 8786 (time 1709056493.73 total 352.13 last 21.19) - open/close 9067 (time 1709056503.74 total 362.14 last 28.08) - open/close 9291 (time 1709056514.36 total 372.76 last 21.10) - open/close 9542 (time 1709056524.86 total 383.26 last 23.91) - open/close 10000 (time 1709056530.13 total 388.53 last 86.80) - open/close 10896 (time 1709056540.14 total 398.54 last 89.54) - open/close 11657 (time 1709056550.15 total 408.55 last 76.02) - open/close 12317 (time 1709056560.16 total 418.56 last 65.95) - open/close 12993 (time 1709056570.17 total 428.57 last 67.55) - open/close 13767 (time 1709056580.17 total 438.57 last 77.39) - open/close 14550 (time 1709056590.17 total 448.57 last 78.25) - open/close 15290 (time 1709056600.18 total 458.58 last 73.96) - open/close 15949 (time 1709056610.19 total 468.59 last 65.82) - open/close 16569 (time 1709056620.20 total 478.60 last 61.93) - open/close 17093 (time 1709056630.21 total 488.61 last 52.35) - open/close 17767 (time 1709056640.22 total 498.62 last 67.37) - open/close 18453 (time 1709056650.23 total 508.63 last 68.54) - open/close 19009 (time 1709056660.26 total 518.65 last 55.44) - open/close 19512 (time 1709056670.26 total 528.66 last 50.28) - open/close 19994 (time 1709056680.26 total 538.66 last 48.18) - open/close 20530 (time 1709056690.27 total 548.67 last 53.55) - open/close 21134 (time 1709056700.28 total 558.68 last 60.34) - open/close 21623 (time 1709056710.30 total 568.70 last 48.82) - open/close 22088 (time 1709056720.31 total 578.71 last 46.46) - open/close 22480 (time 1709056730.33 total 588.73 last 39.11) - open/close 22906 (time 1709056740.34 total 598.74 last 42.55) - open/close 23342 (time 1709056750.36 total 608.76 last 43.52) - open/close 23703 (time 1709056760.39 total 618.79 last 36.01) - open/close 24119 (time 1709056770.41 total 628.81 last 41.51) - open/close 24501 (time 1709056780.43 total 638.83 last 38.12) - open/close 24942 (time 1709056790.45 total 648.85 last 44.00) - open/close 25326 (time 1709056800.46 total 658.86 last 38.38) - open/close 25708 (time 1709056810.47 total 668.87 last 38.16) - open/close 26086 (time 1709056820.48 total 678.88 last 37.74) - open/close 26445 (time 1709056830.50 total 688.90 last 35.83) - open/close 26792 (time 1709056840.52 total 698.92 last 34.64) - open/close 27193 (time 1709056850.58 total 708.98 last 39.89) - open/close 27629 (time 1709056860.59 total 718.99 last 43.52) - open/close 28017 (time 1709056870.60 total 729.00 last 38.76) - open/close 28361 (time 1709056880.61 total 739.01 last 34.38) - open/close 28728 (time 1709056890.64 total 749.04 last 36.59) - open/close 29097 (time 1709056900.65 total 759.05 last 36.88) - open/close 29467 (time 1709056910.67 total 769.07 last 36.90) - open/close 29832 (time 1709056920.68 total 779.08 last 36.48) - open/close 30000 (time 1709056925.03 total 783.43 last 38.57) - open/close 30374 (time 1709056935.05 total 793.45 last 37.34) - open/close 30782 (time 1709056945.14 total 803.54 last 40.44) - open/close 31166 (time 1709056955.15 total 813.55 last 38.37) - open/close 31570 (time 1709056965.15 total 823.55 last 40.40) - open/close 31964 (time 1709056975.16 total 833.56 last 39.34) - open/close 32336 (time 1709056985.18 total 843.58 last 37.14) - open/close 32754 (time 1709056995.19 total 853.59 last 41.76) - open/close 33212 (time 1709057005.20 total 863.60 last 45.76) - open/close 33620 (time 1709057015.21 total 873.61 last 40.75) - open/close 34013 (time 1709057025.23 total 883.63 last 39.23) - open/close 34416 (time 1709057035.25 total 893.65 last 40.20) - open/close 34864 (time 1709057045.26 total 903.66 last 44.77) - open/close 35337 (time 1709057055.28 total 913.68 last 47.23) - open/close 35756 (time 1709057065.30 total 923.70 last 41.80) - open/close 36183 (time 1709057075.31 total 933.71 last 42.66) - open/close 36538 (time 1709057085.32 total 943.72 last 35.46) - open/close 36917 (time 1709057095.33 total 953.73 last 37.88) - open/close 37226 (time 1709057105.35 total 963.75 last 30.82) - open/close 37592 (time 1709057115.38 total 973.78 last 36.51) - open/close 37925 (time 1709057125.40 total 983.80 last 33.23) - open/close 38224 (time 1709057135.40 total 993.80 last 29.89) - open/close 38526 (time 1709057145.43 total 1003.83 last 30.12) - open/close 38928 (time 1709057155.43 total 1013.83 last 40.18) - open/close 39249 (time 1709057165.44 total 1023.84 last 32.09) - open/close 39573 (time 1709057175.44 total 1033.84 last 32.40) - open/close 39891 (time 1709057185.44 total 1043.84 last 31.80) - open/close 40000 (time 1709057189.45 total 1047.85 last 27.18) - open/close 40288 (time 1709057199.48 total 1057.88 last 28.72) - open/close 40662 (time 1709057209.49 total 1067.89 last 37.33) - open/close 40990 (time 1709057219.50 total 1077.90 last 32.78) - open/close 41275 (time 1709057229.52 total 1087.92 last 28.45) - open/close 41612 (time 1709057239.52 total 1097.92 last 33.70) - open/close 41977 (time 1709057249.54 total 1107.94 last 36.43) - open/close 42334 (time 1709057259.56 total 1117.96 last 35.62) - open/close 42656 (time 1709057269.59 total 1127.99 last 32.09) - open/close 42960 (time 1709057279.63 total 1138.03 last 30.29) - open/close 43292 (time 1709057289.63 total 1148.03 last 33.20) - open/close 43616 (time 1709057299.63 total 1158.03 last 32.39) - open/close 44003 (time 1709057309.64 total 1168.04 last 38.67) - open/close 44339 (time 1709057319.65 total 1178.05 last 33.57) - open/close 44672 (time 1709057329.66 total 1188.06 last 33.27) - open/close 44964 (time 1709057339.72 total 1198.12 last 29.01) - open/close 45211 (time 1709057349.73 total 1208.13 last 24.69) - open/close 45544 (time 1709057359.75 total 1218.15 last 33.24) - open/close 45819 (time 1709057369.80 total 1228.20 last 27.34) - open/close 46093 (time 1709057379.84 total 1238.24 last 27.29) - open/close 46345 (time 1709057389.88 total 1248.28 last 25.10) - open/close 46553 (time 1709057399.91 total 1258.31 last 20.75) - open/close 46785 (time 1709057409.94 total 1268.34 last 23.13) - open/close 47048 (time 1709057419.97 total 1278.37 last 26.21) - open/close 47270 (time 1709057429.98 total 1288.38 last 22.17) - open/close 47485 (time 1709057440.02 total 1298.42 last 21.42) - open/close 47731 (time 1709057450.07 total 1308.47 last 24.50) - open/close 47960 (time 1709057460.12 total 1318.52 last 22.77) - open/close 48205 (time 1709057470.15 total 1328.55 last 24.44) - open/close 48448 (time 1709057480.19 total 1338.59 last 24.20) - open/close 48730 (time 1709057490.23 total 1348.63 last 28.08) - open/close 48950 (time 1709057500.26 total 1358.66 last 21.95) - open/close 49174 (time 1709057510.28 total 1368.68 last 22.35) - open/close 49430 (time 1709057520.30 total 1378.70 last 25.53) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1387.11 seconds: 35.78 ops/second - unlinked 0 (time 1709057531 ; total 0 ; last 0) - unlinked 10000 (time 1709057951 ; total 420 ; last 420) - unlinked 20000 (time 1709058280 ; total 749 ; last 329) - unlinked 30000 (time 1709058553 ; total 1022 ; last 273) - unlinked 40000 (time 1709058686 ; total 1155 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 1222 seconds: 40.615383 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server unloading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2