== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:42:15 (1768549335) start mds service on oleg215-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 47 (time 1768549423.78 total 10.89 last 4.32) - open/close 123 (time 1768549433.81 total 20.91 last 7.58) - open/close 364 (time 1768549443.81 total 30.91 last 24.09) - open/close 563 (time 1768549453.84 total 40.94 last 19.84) - open/close 843 (time 1768549464.36 total 51.47 last 26.61) - open/close 1119 (time 1768549474.37 total 61.47 last 27.58) - open/close 1232 (time 1768549484.97 total 72.07 last 10.66) - open/close 1386 (time 1768549494.98 total 82.08 last 15.38) - open/close 1437 (time 1768549505.39 total 92.49 last 4.90) - open/close 1626 (time 1768549516.14 total 103.24 last 17.58) - open/close 1763 (time 1768549526.17 total 113.27 last 13.66) - open/close 1818 (time 1768549536.32 total 123.43 last 5.42) - open/close 1929 (time 1768549546.86 total 133.97 last 10.53) - open/close 2155 (time 1768549556.88 total 143.98 last 22.57) - open/close 2339 (time 1768549566.88 total 153.98 last 18.40) - open/close 2476 (time 1768549576.88 total 163.99 last 13.69) - open/close 2645 (time 1768549587.79 total 174.89 last 15.50) - open/close 2770 (time 1768549597.79 total 184.89 last 12.50) - open/close 2883 (time 1768549608.38 total 195.48 last 10.68) - open/close 2968 (time 1768549618.97 total 206.07 last 8.02) - open/close 3068 (time 1768549630.37 total 217.48 last 8.77) - open/close 3210 (time 1768549640.94 total 228.04 last 13.44) - open/close 3328 (time 1768549651.94 total 239.04 last 10.73) - open/close 3473 (time 1768549662.02 total 249.13 last 14.38) - open/close 3699 (time 1768549672.69 total 259.79 last 21.19) - open/close 3924 (time 1768549683.65 total 270.76 last 20.52) - open/close 4145 (time 1768549694.32 total 281.42 last 20.72) - open/close 4309 (time 1768549704.55 total 291.66 last 16.03) - open/close 4512 (time 1768549714.56 total 301.67 last 20.28) - open/close 4704 (time 1768549724.58 total 311.68 last 19.17) - open/close 4779 (time 1768549734.60 total 321.71 last 7.48) - open/close 5009 (time 1768549744.61 total 331.71 last 22.99) - open/close 5144 (time 1768549755.04 total 342.14 last 12.95) - open/close 5395 (time 1768549765.22 total 352.32 last 24.65) - open/close 5595 (time 1768549775.23 total 362.34 last 19.98) - open/close 5786 (time 1768549785.24 total 372.35 last 19.08) - open/close 5940 (time 1768549795.75 total 382.86 last 14.65) - open/close 6216 (time 1768549806.21 total 393.32 last 26.38) - open/close 6449 (time 1768549816.23 total 403.33 last 23.26) - open/close 6712 (time 1768549826.73 total 413.83 last 25.05) - open/close 6911 (time 1768549837.04 total 424.14 last 19.31) - open/close 7141 (time 1768549847.04 total 434.14 last 23.00) - open/close 7287 (time 1768549858.03 total 445.13 last 13.29) - open/close 7463 (time 1768549868.03 total 455.14 last 17.59) - open/close 7695 (time 1768549878.95 total 466.05 last 21.26) - open/close 7887 (time 1768549889.27 total 476.37 last 18.60) - open/close 8106 (time 1768549899.28 total 486.38 last 21.87) - open/close 8341 (time 1768549909.29 total 496.40 last 23.47) - open/close 8507 (time 1768549920.04 total 507.14 last 15.45) - open/close 8595 (time 1768549930.30 total 517.40 last 8.58) - open/close 8680 (time 1768549940.45 total 527.56 last 8.37) - open/close 8880 (time 1768549950.46 total 537.56 last 19.99) - open/close 9084 (time 1768549961.19 total 548.29 last 19.01) - open/close 9239 (time 1768549972.47 total 559.58 last 13.74) - open/close 9359 (time 1768549983.28 total 570.39 last 11.10) - open/close 9481 (time 1768549994.22 total 581.33 last 11.15) - open/close 9601 (time 1768550005.18 total 592.29 last 10.95) - open/close 9804 (time 1768550015.20 total 602.30 last 20.28) - open/close 9918 (time 1768550025.22 total 612.33 last 11.37) - open/close 10000 (time 1768550028.70 total 615.81 last 23.56) - open/close 10127 (time 1768550039.72 total 626.83 last 11.52) - open/close 10237 (time 1768550049.73 total 636.83 last 10.99) - open/close 10353 (time 1768550059.89 total 647.00 last 11.42) - open/close 10435 (time 1768550070.25 total 657.35 last 7.92) - open/close 10631 (time 1768550081.28 total 668.38 last 17.77) - open/close 10775 (time 1768550091.29 total 678.39 last 14.39) - open/close 10904 (time 1768550101.95 total 689.05 last 12.10) - open/close 11109 (time 1768550112.58 total 699.68 last 19.28) - open/close 11226 (time 1768550123.00 total 710.10 last 11.23) - open/close 11470 (time 1768550133.05 total 720.15 last 24.27) - open/close 11646 (time 1768550143.09 total 730.19 last 17.54) - open/close 11865 (time 1768550153.90 total 741.00 last 20.26) - open/close 11948 (time 1768550166.71 total 753.81 last 6.48) - open/close 12092 (time 1768550177.63 total 764.73 last 13.18) - open/close 12260 (time 1768550187.64 total 774.75 last 16.78) - open/close 12340 (time 1768550198.29 total 785.40 last 7.51) - open/close 12512 (time 1768550208.31 total 795.41 last 17.17) - open/close 12706 (time 1768550218.41 total 805.51 last 19.21) - open/close 12922 (time 1768550228.66 total 815.77 last 21.06) - open/close 13155 (time 1768550238.68 total 825.78 last 23.27) - open/close 13356 (time 1768550249.01 total 836.11 last 19.45) - open/close 13525 (time 1768550261.25 total 848.36 last 13.80) - open/close 13679 (time 1768550271.28 total 858.38 last 15.36) - open/close 13795 (time 1768550283.33 total 870.43 last 9.63) - open/close 13967 (time 1768550293.34 total 880.44 last 17.19) - open/close 14033 (time 1768550304.19 total 891.29 last 6.08) - open/close 14160 (time 1768550314.21 total 901.31 last 12.67) - open/close 14394 (time 1768550326.96 total 914.06 last 18.35) - open/close 14621 (time 1768550336.97 total 924.07 last 22.69) - open/close 14811 (time 1768550346.98 total 934.09 last 18.97) - open/close 14987 (time 1768550356.99 total 944.10 last 17.58) - open/close 15106 (time 1768550367.00 total 954.11 last 11.89) - open/close 15273 (time 1768550377.01 total 964.11 last 16.70) - open/close 15500 (time 1768550387.02 total 974.12 last 22.67) - open/close 15703 (time 1768550400.13 total 987.24 last 15.48) - open/close 15825 (time 1768550411.11 total 998.22 last 11.11) - open/close 16029 (time 1768550422.07 total 1009.17 last 18.62) - open/close 16209 (time 1768550432.08 total 1019.19 last 17.98) - open/close 16392 (time 1768550442.12 total 1029.23 last 18.23) - open/close 16548 (time 1768550452.35 total 1039.45 last 15.26) - open/close 16773 (time 1768550463.02 total 1050.12 last 21.08) - open/close 16964 (time 1768550473.05 total 1060.15 last 19.05) - open/close 17622 (time 1768550483.07 total 1070.17 last 65.65) - open/close 18456 (time 1768550493.08 total 1080.18 last 83.32) - open/close 19211 (time 1768550503.08 total 1090.19 last 75.46) - open/close 19937 (time 1768550513.09 total 1100.20 last 72.53) - open/close 20694 (time 1768550523.10 total 1110.20 last 75.67) - open/close 21515 (time 1768550533.12 total 1120.23 last 81.88) - open/close 22300 (time 1768550543.12 total 1130.23 last 78.50) - open/close 23195 (time 1768550553.13 total 1140.23 last 89.43) - open/close 24006 (time 1768550563.14 total 1150.25 last 81.00) - open/close 24879 (time 1768550573.15 total 1160.25 last 87.26) - open/close 25765 (time 1768550583.17 total 1170.27 last 88.42) - open/close 26445 (time 1768550593.18 total 1180.28 last 67.94) - open/close 27206 (time 1768550603.19 total 1190.29 last 76.02) - open/close 28006 (time 1768550613.19 total 1200.29 last 79.98) - open/close 28803 (time 1768550623.20 total 1210.30 last 79.65) - open/close 29621 (time 1768550633.20 total 1220.30 last 81.78) - open/close 30000 (time 1768550638.51 total 1225.61 last 71.42) - open/close 30781 (time 1768550648.51 total 1235.61 last 78.08) - open/close 31642 (time 1768550658.51 total 1245.62 last 86.04) - open/close 32523 (time 1768550668.53 total 1255.63 last 88.00) - open/close 33353 (time 1768550678.53 total 1265.64 last 82.93) - open/close 34139 (time 1768550688.54 total 1275.64 last 78.58) - open/close 35045 (time 1768550698.54 total 1285.64 last 90.58) - open/close 36005 (time 1768550708.55 total 1295.65 last 95.92) - open/close 36891 (time 1768550718.55 total 1305.66 last 88.56) - open/close 37643 (time 1768550728.57 total 1315.67 last 75.10) - open/close 38394 (time 1768550738.57 total 1325.67 last 75.09) - open/close 39192 (time 1768550748.57 total 1335.68 last 79.76) - open/close 40000 (time 1768550758.09 total 1345.19 last 84.91) - open/close 40840 (time 1768550768.09 total 1355.20 last 83.97) - open/close 41642 (time 1768550778.10 total 1365.20 last 80.17) - open/close 42533 (time 1768550788.11 total 1375.21 last 88.99) - open/close 43305 (time 1768550798.11 total 1385.22 last 77.18) - open/close 44080 (time 1768550808.12 total 1395.22 last 77.44) - open/close 44849 (time 1768550818.17 total 1405.27 last 76.54) - open/close 45623 (time 1768550828.17 total 1415.28 last 77.36) - open/close 46364 (time 1768550838.18 total 1425.29 last 74.02) - open/close 47122 (time 1768550848.19 total 1435.29 last 75.76) - open/close 47869 (time 1768550858.19 total 1445.30 last 74.65) - open/close 48498 (time 1768550868.20 total 1455.30 last 62.86) - open/close 49323 (time 1768550878.22 total 1465.32 last 82.37) total: 50000 open/close in 1473.04 seconds: 33.94 ops/second - unlinked 0 (time 1768550890 ; total 0 ; last 0) - unlinked 10000 (time 1768551014 ; total 124 ; last 124) - unlinked 20000 (time 1768551158 ; total 268 ; last 144) - unlinked 30000 (time 1768551300 ; total 410 ; last 142) - unlinked 40000 (time 1768551434 ; total 544 ; last 134) total: 50000 unlinks in 694 seconds: 72.046112 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg215-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2