== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:17:20 (1786817840) start mds service on oleg104-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg104-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg104-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 297 (time 1786817949.08 total 10.40 last 28.57) - open/close 560 (time 1786817960.90 total 22.22 last 22.25) - open/close 784 (time 1786817971.89 total 33.21 last 20.37) - open/close 1066 (time 1786817981.90 total 43.22 last 28.17) - open/close 1255 (time 1786817992.37 total 53.69 last 18.06) - open/close 1484 (time 1786818003.01 total 64.33 last 21.51) - open/close 1699 (time 1786818013.04 total 74.36 last 21.45) - open/close 1869 (time 1786818023.98 total 85.30 last 15.53) - open/close 2079 (time 1786818034.94 total 96.26 last 19.16) - open/close 2236 (time 1786818045.24 total 106.56 last 15.24) - open/close 2431 (time 1786818056.05 total 117.37 last 18.05) - open/close 2629 (time 1786818066.88 total 128.20 last 18.27) - open/close 2757 (time 1786818076.89 total 138.21 last 12.79) - open/close 2902 (time 1786818087.23 total 148.54 last 14.03) - open/close 3081 (time 1786818097.66 total 158.98 last 17.16) - open/close 3254 (time 1786818108.11 total 169.43 last 16.55) - open/close 3497 (time 1786818118.11 total 179.43 last 24.30) - open/close 3776 (time 1786818128.13 total 189.45 last 27.86) - open/close 4082 (time 1786818138.68 total 200.00 last 29.01) - open/close 4340 (time 1786818149.49 total 210.81 last 23.86) - open/close 4539 (time 1786818160.43 total 221.75 last 18.19) - open/close 4810 (time 1786818171.89 total 233.21 last 23.65) - open/close 5114 (time 1786818182.83 total 244.15 last 27.80) - open/close 5387 (time 1786818193.23 total 254.54 last 26.25) - open/close 5599 (time 1786818203.96 total 265.28 last 19.75) - open/close 5847 (time 1786818214.33 total 275.65 last 23.91) - open/close 6129 (time 1786818224.70 total 286.02 last 27.20) - open/close 6398 (time 1786818234.75 total 296.07 last 26.78) - open/close 6693 (time 1786818245.50 total 306.82 last 27.44) - open/close 6975 (time 1786818256.37 total 317.69 last 25.93) - open/close 7237 (time 1786818267.00 total 328.32 last 24.64) - open/close 7465 (time 1786818277.56 total 338.88 last 21.60) - open/close 7690 (time 1786818290.34 total 351.65 last 17.61) - open/close 7964 (time 1786818300.34 total 361.66 last 27.39) - open/close 8235 (time 1786818310.34 total 371.66 last 27.10) - open/close 8479 (time 1786818320.51 total 381.83 last 24.00) - open/close 8810 (time 1786818330.52 total 391.84 last 33.05) - open/close 9098 (time 1786818340.53 total 401.85 last 28.76) - open/close 9357 (time 1786818350.72 total 412.03 last 25.44) - open/close 9454 (time 1786818361.28 total 422.60 last 9.18) - open/close 9520 (time 1786818371.29 total 432.61 last 6.59) - open/close 9765 (time 1786818381.63 total 442.95 last 23.70) - open/close 9981 (time 1786818391.64 total 452.96 last 21.59) - open/close 10184 (time 1786818401.64 total 462.96 last 20.29) - open/close 10407 (time 1786818411.96 total 473.28 last 21.62) - open/close 10575 (time 1786818422.70 total 484.02 last 15.63) - open/close 10873 (time 1786818433.67 total 494.99 last 27.17) - open/close 11170 (time 1786818443.68 total 505.00 last 29.67) - open/close 11436 (time 1786818454.29 total 515.61 last 25.08) - open/close 11703 (time 1786818464.29 total 525.61 last 26.68) - open/close 11948 (time 1786818474.30 total 535.62 last 24.49) - open/close 12244 (time 1786818485.16 total 546.48 last 27.26) - open/close 12411 (time 1786818495.47 total 556.78 last 16.20) - open/close 12671 (time 1786818506.24 total 567.56 last 24.12) - open/close 12928 (time 1786818516.25 total 577.57 last 25.68) - open/close 13065 (time 1786818526.33 total 587.65 last 13.60) - open/close 13289 (time 1786818536.33 total 597.65 last 22.40) - open/close 13517 (time 1786818546.87 total 608.19 last 21.62) - open/close 13614 (time 1786818556.91 total 618.23 last 9.66) - open/close 13857 (time 1786818567.15 total 628.47 last 23.72) - open/close 14056 (time 1786818577.17 total 638.49 last 19.87) - open/close 14223 (time 1786818587.33 total 648.65 last 16.43) - open/close 14487 (time 1786818597.34 total 658.66 last 26.38) - open/close 14733 (time 1786818607.94 total 669.26 last 23.21) - open/close 14961 (time 1786818618.96 total 680.28 last 20.68) - open/close 15058 (time 1786818628.97 total 690.29 last 9.69) - open/close 15278 (time 1786818640.01 total 701.33 last 19.92) - open/close 15553 (time 1786818650.02 total 711.34 last 27.49) - open/close 15823 (time 1786818661.03 total 722.35 last 24.52) - open/close 16103 (time 1786818671.04 total 732.36 last 27.97) - open/close 16367 (time 1786818681.06 total 742.38 last 26.35) - open/close 16634 (time 1786818691.78 total 753.10 last 24.89) - open/close 16837 (time 1786818702.67 total 763.99 last 18.65) - open/close 17044 (time 1786818712.79 total 774.11 last 20.45) - open/close 17184 (time 1786818722.80 total 784.12 last 13.98) - open/close 17376 (time 1786818733.64 total 794.96 last 17.71) - open/close 17610 (time 1786818744.25 total 805.57 last 22.06) - open/close 17827 (time 1786818755.12 total 816.44 last 19.97) - open/close 18116 (time 1786818765.64 total 826.96 last 27.48) - open/close 18309 (time 1786818777.04 total 838.36 last 16.93) - open/close 18550 (time 1786818787.05 total 848.37 last 24.08) - open/close 18854 (time 1786818797.11 total 858.43 last 30.21) - open/close 18910 (time 1786818807.65 total 868.97 last 5.32) - open/close 19195 (time 1786818817.66 total 878.98 last 28.45) - open/close 19392 (time 1786818828.28 total 889.60 last 18.56) - open/close 19592 (time 1786818839.29 total 900.61 last 18.16) - open/close 19854 (time 1786818850.17 total 911.49 last 24.07) - open/close 20000 (time 1786818857.89 total 919.21 last 18.91) - open/close 20158 (time 1786818867.90 total 929.22 last 15.78) - open/close 20426 (time 1786818878.85 total 940.17 last 24.48) - open/close 20615 (time 1786818888.87 total 950.19 last 18.87) - open/close 20843 (time 1786818899.09 total 960.41 last 22.31) - open/close 21023 (time 1786818909.13 total 970.45 last 17.92) - open/close 21114 (time 1786818919.14 total 980.46 last 9.09) - open/close 21990 (time 1786818929.15 total 990.47 last 87.49) - open/close 22917 (time 1786818939.15 total 1000.47 last 92.68) - open/close 23986 (time 1786818949.16 total 1010.48 last 106.87) - open/close 24813 (time 1786818959.18 total 1020.50 last 82.51) - open/close 25714 (time 1786818969.18 total 1030.50 last 90.07) - open/close 26558 (time 1786818979.19 total 1040.51 last 84.36) - open/close 27563 (time 1786818989.20 total 1050.52 last 100.40) - open/close 28536 (time 1786818999.21 total 1060.53 last 97.14) - open/close 29318 (time 1786819009.22 total 1070.54 last 78.16) - open/close 30000 (time 1786819016.73 total 1078.05 last 90.83) - open/close 30976 (time 1786819026.74 total 1088.06 last 97.50) - open/close 31826 (time 1786819036.74 total 1098.06 last 84.98) - open/close 32657 (time 1786819046.75 total 1108.07 last 83.05) - open/close 33588 (time 1786819056.77 total 1118.09 last 92.88) - open/close 34556 (time 1786819066.78 total 1128.09 last 96.76) - open/close 35540 (time 1786819076.78 total 1138.10 last 98.37) - open/close 36505 (time 1786819086.79 total 1148.11 last 96.37) - open/close 37400 (time 1786819096.80 total 1158.12 last 89.41) - open/close 38266 (time 1786819106.81 total 1168.12 last 86.57) - open/close 39205 (time 1786819116.82 total 1178.14 last 93.79) - open/close 39965 (time 1786819126.82 total 1188.14 last 75.94) - open/close 40722 (time 1786819136.84 total 1198.16 last 75.61) - open/close 41759 (time 1786819146.84 total 1208.16 last 103.63) - open/close 42724 (time 1786819156.85 total 1218.17 last 96.44) - open/close 43479 (time 1786819166.85 total 1228.17 last 75.46) - open/close 44207 (time 1786819176.86 total 1238.18 last 72.78) - open/close 44939 (time 1786819186.86 total 1248.18 last 73.20) - open/close 45860 (time 1786819196.87 total 1258.19 last 91.99) - open/close 46541 (time 1786819206.87 total 1268.19 last 68.09) - open/close 47315 (time 1786819216.87 total 1278.19 last 77.39) - open/close 48093 (time 1786819226.89 total 1288.21 last 77.65) - open/close 48853 (time 1786819236.90 total 1298.22 last 75.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1308.16 seconds: 37.94 ops/second - unlinked 0 (time 1786819251 ; total 0 ; last 0) - unlinked 10000 (time 1786819367 ; total 116 ; last 116) - unlinked 20000 (time 1786819501 ; total 250 ; last 134) - unlinked 30000 (time 1786819577 ; total 326 ; last 76) - unlinked 40000 (time 1786819646 ; total 395 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 460 seconds: 107.893478 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2