== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:50:46 (1786751446) start mds service on oleg102-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg102-server: detected 4 online CPUs by sysfs oleg102-server: MODOPTS_LIBCFS= oleg102-server: Force libcfs to create 2 CPU partitions oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg102-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg102-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 161 (time 1786751575.65 total 10.81 last 14.89) - open/close 290 (time 1786751586.39 total 21.56 last 12.01) - open/close 366 (time 1786751596.65 total 31.81 last 7.41) - open/close 498 (time 1786751606.65 total 41.82 last 13.20) - open/close 591 (time 1786751617.31 total 52.48 last 8.72) - open/close 882 (time 1786751628.13 total 63.30 last 26.90) - open/close 998 (time 1786751638.14 total 73.31 last 11.59) - open/close 1231 (time 1786751648.15 total 83.31 last 23.28) - open/close 1413 (time 1786751658.75 total 93.92 last 17.16) - open/close 1447 (time 1786751668.76 total 103.93 last 3.40) - open/close 1634 (time 1786751679.60 total 114.77 last 17.26) - open/close 1785 (time 1786751690.16 total 125.32 last 14.30) - open/close 1914 (time 1786751700.16 total 135.33 last 12.90) - open/close 2058 (time 1786751710.48 total 145.65 last 13.95) - open/close 2299 (time 1786751721.34 total 156.51 last 22.19) - open/close 2386 (time 1786751732.11 total 167.28 last 8.08) - open/close 2560 (time 1786751742.36 total 177.53 last 16.98) - open/close 2814 (time 1786751752.37 total 187.54 last 25.37) - open/close 2991 (time 1786751763.25 total 198.42 last 16.28) - open/close 3126 (time 1786751774.19 total 209.36 last 12.34) - open/close 3360 (time 1786751784.21 total 219.38 last 23.35) - open/close 3557 (time 1786751795.11 total 230.27 last 18.08) - open/close 3700 (time 1786751805.35 total 240.51 last 13.97) - open/close 3736 (time 1786751815.55 total 250.71 last 3.53) - open/close 3929 (time 1786751825.55 total 260.72 last 19.29) - open/close 4168 (time 1786751837.21 total 272.38 last 20.50) - open/close 4459 (time 1786751847.71 total 282.88 last 27.71) - open/close 4631 (time 1786751858.65 total 293.82 last 15.73) - open/close 4747 (time 1786751869.59 total 304.76 last 10.60) - open/close 4835 (time 1786751879.60 total 314.77 last 8.79) - open/close 4986 (time 1786751890.29 total 325.46 last 14.13) - open/close 5064 (time 1786751900.31 total 335.48 last 7.79) - open/close 5200 (time 1786751910.55 total 345.72 last 13.27) - open/close 5424 (time 1786751920.55 total 355.72 last 22.40) - open/close 5593 (time 1786751931.22 total 366.39 last 15.84) - open/close 5740 (time 1786751941.67 total 376.83 last 14.08) - open/close 5858 (time 1786751953.57 total 388.74 last 9.91) - open/close 6082 (time 1786751963.99 total 399.16 last 21.50) - open/close 6296 (time 1786751974.30 total 409.47 last 20.75) - open/close 6485 (time 1786751986.60 total 421.76 last 15.37) - open/close 6668 (time 1786752001.88 total 437.05 last 11.97) - open/close 6835 (time 1786752012.84 total 448.01 last 15.24) - open/close 7104 (time 1786752023.53 total 458.69 last 25.17) - open/close 7229 (time 1786752033.54 total 468.70 last 12.49) - open/close 7420 (time 1786752043.56 total 478.73 last 19.06) - open/close 7591 (time 1786752054.46 total 489.63 last 15.68) - open/close 7805 (time 1786752064.48 total 499.64 last 21.37) - open/close 7873 (time 1786752075.37 total 510.54 last 6.24) - open/close 8044 (time 1786752087.41 total 522.58 last 14.20) - open/close 8265 (time 1786752097.41 total 532.58 last 22.09) - open/close 8347 (time 1786752107.54 total 542.71 last 8.10) - open/close 8478 (time 1786752118.18 total 553.35 last 12.31) - open/close 8735 (time 1786752129.11 total 564.28 last 23.52) - open/close 9013 (time 1786752139.12 total 574.29 last 27.77) - open/close 9167 (time 1786752149.41 total 584.58 last 14.96) - open/close 9343 (time 1786752159.43 total 594.60 last 17.56) - open/close 9516 (time 1786752170.29 total 605.46 last 15.94) - open/close 9682 (time 1786752182.85 total 618.01 last 13.22) - open/close 9959 (time 1786752192.86 total 628.03 last 27.65) - open/close 10196 (time 1786752203.23 total 638.40 last 22.85) - open/close 10456 (time 1786752213.24 total 648.40 last 25.99) - open/close 10638 (time 1786752223.78 total 658.95 last 17.26) - open/close 10833 (time 1786752234.02 total 669.19 last 19.04) - open/close 11039 (time 1786752244.69 total 679.86 last 19.31) - open/close 11138 (time 1786752255.57 total 690.74 last 9.10) - open/close 11311 (time 1786752265.84 total 701.01 last 16.85) - open/close 11590 (time 1786752275.85 total 711.02 last 27.87) - open/close 11701 (time 1786752286.69 total 721.86 last 10.24) - open/close 11976 (time 1786752297.26 total 732.42 last 26.03) - open/close 12225 (time 1786752308.33 total 743.50 last 22.49) - open/close 12357 (time 1786752318.35 total 753.51 last 13.18) - open/close 12553 (time 1786752328.36 total 763.53 last 19.56) - open/close 12793 (time 1786752338.47 total 773.64 last 23.75) - open/close 13053 (time 1786752349.35 total 784.52 last 23.90) - open/close 13293 (time 1786752359.36 total 794.52 last 23.98) - open/close 13558 (time 1786752369.36 total 804.53 last 26.50) - open/close 13708 (time 1786752380.31 total 815.48 last 13.70) - open/close 13913 (time 1786752390.71 total 825.88 last 19.70) - open/close 14145 (time 1786752400.74 total 835.91 last 23.14) - open/close 14388 (time 1786752410.85 total 846.02 last 24.03) - open/close 14626 (time 1786752420.85 total 856.02 last 23.79) - open/close 14875 (time 1786752431.52 total 866.68 last 23.35) - open/close 15083 (time 1786752441.53 total 876.69 last 20.78) - open/close 15297 (time 1786752451.53 total 886.70 last 21.38) - open/close 15544 (time 1786752461.98 total 897.15 last 23.65) - open/close 15720 (time 1786752471.98 total 907.15 last 17.59) - open/close 15954 (time 1786752481.99 total 917.16 last 23.39) - open/close 16189 (time 1786752492.00 total 927.17 last 23.47) - open/close 16458 (time 1786752502.82 total 937.99 last 24.86) - open/close 16684 (time 1786752512.83 total 948.00 last 22.58) - open/close 16883 (time 1786752522.83 total 958.00 last 19.90) - open/close 17139 (time 1786752532.85 total 968.02 last 25.55) - open/close 17323 (time 1786752543.16 total 978.32 last 17.85) - open/close 17516 (time 1786752553.16 total 988.33 last 19.29) - open/close 17781 (time 1786752565.47 total 1000.64 last 21.52) - open/close 18019 (time 1786752576.16 total 1011.32 last 22.28) - open/close 18163 (time 1786752586.36 total 1021.53 last 14.11) - open/close 18339 (time 1786752596.37 total 1031.54 last 17.59) - open/close 18985 (time 1786752606.38 total 1041.55 last 64.52) - open/close 19842 (time 1786752616.40 total 1051.56 last 85.57) - open/close 20000 (time 1786752618.89 total 1054.06 last 63.32) - open/close 20646 (time 1786752628.90 total 1064.07 last 64.52) - open/close 21273 (time 1786752638.91 total 1074.08 last 62.64) - open/close 22086 (time 1786752648.94 total 1084.11 last 81.11) - open/close 22910 (time 1786752658.94 total 1094.11 last 82.35) - open/close 23842 (time 1786752668.95 total 1104.11 last 93.19) - open/close 24768 (time 1786752678.95 total 1114.12 last 92.54) - open/close 25659 (time 1786752688.95 total 1124.12 last 89.08) - open/close 26642 (time 1786752698.96 total 1134.13 last 98.24) - open/close 27521 (time 1786752708.97 total 1144.14 last 87.81) - open/close 28255 (time 1786752718.97 total 1154.14 last 73.37) - open/close 29119 (time 1786752728.98 total 1164.15 last 86.33) - open/close 29965 (time 1786752739.00 total 1174.17 last 84.47) - open/close 30949 (time 1786752749.01 total 1184.17 last 98.33) - open/close 31737 (time 1786752759.02 total 1194.19 last 78.68) - open/close 32559 (time 1786752769.03 total 1204.20 last 82.12) - open/close 33427 (time 1786752779.04 total 1214.21 last 86.72) - open/close 34154 (time 1786752789.06 total 1224.22 last 72.58) - open/close 34877 (time 1786752799.08 total 1234.24 last 72.16) - open/close 35683 (time 1786752809.09 total 1244.26 last 80.49) - open/close 36432 (time 1786752819.09 total 1254.26 last 74.89) - open/close 37210 (time 1786752829.09 total 1264.26 last 77.78) - open/close 38093 (time 1786752839.10 total 1274.27 last 88.24) - open/close 39031 (time 1786752849.11 total 1284.27 last 93.74) - open/close 39983 (time 1786752859.11 total 1294.27 last 95.20) - open/close 40829 (time 1786752869.11 total 1304.28 last 84.59) - open/close 41687 (time 1786752879.11 total 1314.28 last 85.80) - open/close 42472 (time 1786752889.11 total 1324.28 last 78.48) - open/close 43334 (time 1786752899.12 total 1334.29 last 86.10) - open/close 44174 (time 1786752909.13 total 1344.29 last 83.97) - open/close 45000 (time 1786752919.13 total 1354.29 last 82.59) - open/close 45782 (time 1786752929.13 total 1364.30 last 78.18) - open/close 46359 (time 1786752939.13 total 1374.30 last 57.66) - open/close 47094 (time 1786752949.14 total 1384.31 last 73.48) - open/close 47950 (time 1786752959.15 total 1394.32 last 85.49) - open/close 48762 (time 1786752969.16 total 1404.33 last 81.12) - open/close 49430 (time 1786752979.16 total 1414.33 last 66.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1417.39 seconds: 35.02 ops/second - unlinked 0 (time 1786752987 ; total 0 ; last 0) - unlinked 10000 (time 1786753116 ; total 129 ; last 129) - unlinked 20000 (time 1786753236 ; total 249 ; last 120) - unlinked 30000 (time 1786753358 ; total 371 ; last 122) - unlinked 40000 (time 1786753473 ; total 486 ; last 115) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 600 seconds: 82.718330 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg102-server unloading modules via unload_modules_local on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: modules unloaded. pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2