== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:58:34 (1787857114) start mds service on oleg331-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1787857208.75 total 10.01 last 22.07) - open/close 416 (time 1787857218.76 total 20.03 last 19.47) - open/close 608 (time 1787857231.61 total 32.87 last 14.95) - open/close 856 (time 1787857244.42 total 45.68 last 19.36) - open/close 1120 (time 1787857255.03 total 56.30 last 24.88) - open/close 1355 (time 1787857265.66 total 66.92 last 22.11) - open/close 1580 (time 1787857275.68 total 76.94 last 22.46) - open/close 1805 (time 1787857286.14 total 87.40 last 21.51) - open/close 2068 (time 1787857296.14 total 97.40 last 26.30) - open/close 2326 (time 1787857306.14 total 107.40 last 25.80) - open/close 2501 (time 1787857317.13 total 118.39 last 15.93) - open/close 2702 (time 1787857327.13 total 128.40 last 20.08) - open/close 2833 (time 1787857337.82 total 139.09 last 12.26) - open/close 3098 (time 1787857347.85 total 149.11 last 26.43) - open/close 3323 (time 1787857358.34 total 159.61 last 21.44) - open/close 3564 (time 1787857368.76 total 170.02 last 23.14) - open/close 3808 (time 1787857379.33 total 180.59 last 23.07) - open/close 3954 (time 1787857389.96 total 191.22 last 13.74) - open/close 4174 (time 1787857400.53 total 201.80 last 20.80) - open/close 4396 (time 1787857411.20 total 212.46 last 20.81) - open/close 4550 (time 1787857421.21 total 222.48 last 15.38) - open/close 4666 (time 1787857432.13 total 233.39 last 10.63) - open/close 4820 (time 1787857442.13 total 243.40 last 15.39) - open/close 5036 (time 1787857452.14 total 253.40 last 21.60) - open/close 5294 (time 1787857462.14 total 263.40 last 25.79) - open/close 5499 (time 1787857472.19 total 273.45 last 20.40) - open/close 5753 (time 1787857482.64 total 283.90 last 24.31) - open/close 5994 (time 1787857493.22 total 294.49 last 22.76) - open/close 6219 (time 1787857503.57 total 304.83 last 21.75) - open/close 6345 (time 1787857513.61 total 314.87 last 12.55) - open/close 6448 (time 1787857523.68 total 324.94 last 10.23) - open/close 6618 (time 1787857533.70 total 334.96 last 16.96) - open/close 6734 (time 1787857543.75 total 345.01 last 11.54) - open/close 6966 (time 1787857554.37 total 355.63 last 21.84) - open/close 7193 (time 1787857564.94 total 366.20 last 21.48) - open/close 7439 (time 1787857575.94 total 377.20 last 22.37) - open/close 7687 (time 1787857585.94 total 387.21 last 24.78) - open/close 7909 (time 1787857596.10 total 397.36 last 21.87) - open/close 8136 (time 1787857606.84 total 408.11 last 21.13) - open/close 8340 (time 1787857616.86 total 418.12 last 20.37) - open/close 8429 (time 1787857626.86 total 428.13 last 8.90) - open/close 8675 (time 1787857636.89 total 438.15 last 24.54) - open/close 8911 (time 1787857646.90 total 448.17 last 23.56) - open/close 9056 (time 1787857657.42 total 458.68 last 13.79) - open/close 9248 (time 1787857667.42 total 468.69 last 19.19) - open/close 9427 (time 1787857678.37 total 479.63 last 16.35) - open/close 9607 (time 1787857688.46 total 489.73 last 17.84) - open/close 9853 (time 1787857699.32 total 500.58 last 22.66) - open/close 10000 (time 1787857706.34 total 507.60 last 20.94) - open/close 10244 (time 1787857716.92 total 518.19 last 23.05) - open/close 10471 (time 1787857727.17 total 528.43 last 22.15) - open/close 10746 (time 1787857737.19 total 538.45 last 27.45) - open/close 10950 (time 1787857747.41 total 548.67 last 19.96) - open/close 11208 (time 1787857758.47 total 559.73 last 23.34) - open/close 11338 (time 1787857768.59 total 569.85 last 12.84) - open/close 11454 (time 1787857778.88 total 580.15 last 11.26) - open/close 11654 (time 1787857789.64 total 590.91 last 18.59) - open/close 11923 (time 1787857801.73 total 603.00 last 22.25) - open/close 12126 (time 1787857811.74 total 613.00 last 20.28) - open/close 12291 (time 1787857823.17 total 624.44 last 14.43) - open/close 12469 (time 1787857833.18 total 634.44 last 17.79) - open/close 12641 (time 1787857843.19 total 644.45 last 17.19) - open/close 12779 (time 1787857854.22 total 655.49 last 12.51) - open/close 12973 (time 1787857864.40 total 665.66 last 19.07) - open/close 13218 (time 1787857874.40 total 675.67 last 24.49) - open/close 13312 (time 1787857884.71 total 685.97 last 9.12) - open/close 13397 (time 1787857895.10 total 696.36 last 8.18) - open/close 13598 (time 1787857906.05 total 707.31 last 18.36) - open/close 13766 (time 1787857916.08 total 717.34 last 16.75) - open/close 13894 (time 1787857926.09 total 727.35 last 12.79) - open/close 14100 (time 1787857936.10 total 737.36 last 20.58) - open/close 14315 (time 1787857948.82 total 750.09 last 16.89) - open/close 14518 (time 1787857958.84 total 760.10 last 20.28) - open/close 14646 (time 1787857969.68 total 770.94 last 11.80) - open/close 14749 (time 1787857980.17 total 781.43 last 9.82) - open/close 14885 (time 1787857990.61 total 791.87 last 13.03) - open/close 15129 (time 1787858000.62 total 801.89 last 24.37) - open/close 15234 (time 1787858011.41 total 812.67 last 9.73) - open/close 15365 (time 1787858021.44 total 822.71 last 13.05) - open/close 15502 (time 1787858031.77 total 833.03 last 13.27) - open/close 15620 (time 1787858041.98 total 843.24 last 11.56) - open/close 15704 (time 1787858051.99 total 853.26 last 8.39) - open/close 15768 (time 1787858062.92 total 864.18 last 5.86) - open/close 15898 (time 1787858073.61 total 874.88 last 12.16) - open/close 16054 (time 1787858083.62 total 884.88 last 15.59) - open/close 16312 (time 1787858093.89 total 895.16 last 25.11) - open/close 16554 (time 1787858103.90 total 905.16 last 24.19) - open/close 16791 (time 1787858114.52 total 915.78 last 22.32) - open/close 16936 (time 1787858125.94 total 927.21 last 12.69) - open/close 17092 (time 1787858136.46 total 937.72 last 14.84) - open/close 17152 (time 1787858147.22 total 948.48 last 5.58) - open/close 17262 (time 1787858158.09 total 959.35 last 10.12) - open/close 17391 (time 1787858168.11 total 969.37 last 12.87) - open/close 17954 (time 1787858178.11 total 979.37 last 56.29) - open/close 18574 (time 1787858188.13 total 989.39 last 61.90) - open/close 19237 (time 1787858198.14 total 999.40 last 66.20) - open/close 19927 (time 1787858208.14 total 1009.40 last 69.00) - open/close 20586 (time 1787858218.16 total 1019.42 last 65.77) - open/close 21235 (time 1787858228.16 total 1029.43 last 64.88) - open/close 21885 (time 1787858238.18 total 1039.44 last 64.92) - open/close 22509 (time 1787858248.20 total 1049.46 last 62.24) - open/close 23130 (time 1787858258.21 total 1059.47 last 62.07) - open/close 23834 (time 1787858268.21 total 1069.47 last 70.37) - open/close 24485 (time 1787858278.22 total 1079.49 last 65.02) - open/close 25169 (time 1787858288.23 total 1089.49 last 68.36) - open/close 25902 (time 1787858298.24 total 1099.50 last 73.22) - open/close 26588 (time 1787858308.25 total 1109.51 last 68.55) - open/close 27232 (time 1787858318.25 total 1119.51 last 64.39) - open/close 27883 (time 1787858328.26 total 1129.53 last 64.99) - open/close 28495 (time 1787858338.27 total 1139.53 last 61.18) - open/close 29156 (time 1787858348.28 total 1149.54 last 66.02) - open/close 29825 (time 1787858358.29 total 1159.56 last 66.81) - open/close 30000 (time 1787858360.98 total 1162.24 last 65.17) - open/close 30648 (time 1787858370.98 total 1172.24 last 64.80) - open/close 31366 (time 1787858380.99 total 1182.25 last 71.75) - open/close 32024 (time 1787858391.00 total 1192.26 last 65.74) - open/close 32738 (time 1787858401.01 total 1202.27 last 71.30) - open/close 33423 (time 1787858411.01 total 1212.28 last 68.47) - open/close 34085 (time 1787858421.02 total 1222.28 last 66.15) - open/close 34679 (time 1787858431.04 total 1232.30 last 59.30) - open/close 35332 (time 1787858441.05 total 1242.31 last 65.25) - open/close 35986 (time 1787858451.05 total 1252.31 last 65.39) - open/close 36622 (time 1787858461.07 total 1262.34 last 63.43) - open/close 37236 (time 1787858471.08 total 1272.34 last 61.39) - open/close 37901 (time 1787858481.09 total 1282.35 last 66.42) - open/close 38494 (time 1787858491.09 total 1292.36 last 59.27) - open/close 39137 (time 1787858501.10 total 1302.36 last 64.26) - open/close 39817 (time 1787858511.11 total 1312.37 last 67.92) - open/close 40000 (time 1787858513.80 total 1315.06 last 68.06) - open/close 40666 (time 1787858523.81 total 1325.07 last 66.57) - open/close 41304 (time 1787858533.81 total 1335.07 last 63.78) - open/close 41997 (time 1787858543.82 total 1345.08 last 69.25) - open/close 42683 (time 1787858553.83 total 1355.09 last 68.52) - open/close 43327 (time 1787858563.83 total 1365.09 last 64.37) - open/close 43898 (time 1787858573.84 total 1375.10 last 57.08) - open/close 44453 (time 1787858583.85 total 1385.11 last 55.45) - open/close 45043 (time 1787858593.85 total 1395.11 last 58.99) - open/close 45675 (time 1787858603.87 total 1405.13 last 63.05) - open/close 46327 (time 1787858613.88 total 1415.15 last 65.11) - open/close 46981 (time 1787858623.91 total 1425.17 last 65.25) - open/close 47624 (time 1787858633.91 total 1435.18 last 64.26) - open/close 48174 (time 1787858643.91 total 1445.18 last 54.99) - open/close 48792 (time 1787858653.93 total 1455.19 last 61.73) - open/close 49479 (time 1787858663.94 total 1465.20 last 68.62) total: 50000 open/close in 1473.14 seconds: 33.94 ops/second - unlinked 0 (time 1787858676 ; total 0 ; last 0) - unlinked 10000 (time 1787858832 ; total 156 ; last 156) - unlinked 20000 (time 1787858994 ; total 318 ; last 162) - unlinked 30000 (time 1787859152 ; total 476 ; last 158) - unlinked 40000 (time 1787859302 ; total 626 ; last 150) total: 50000 unlinks in 769 seconds: 65.019508 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2