== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:12:05 (1779207125) start mds service on oleg229-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 132 (time 1779207244.93 total 10.39 last 12.70) - open/close 269 (time 1779207254.94 total 20.41 last 13.68) - open/close 513 (time 1779207265.14 total 30.61 last 23.92) - open/close 711 (time 1779207275.65 total 41.11 last 18.84) - open/close 872 (time 1779207285.66 total 51.13 last 16.08) - open/close 1129 (time 1779207296.07 total 61.54 last 24.68) - open/close 1243 (time 1779207306.07 total 71.54 last 11.40) - open/close 1409 (time 1779207317.10 total 82.57 last 15.05) - open/close 1633 (time 1779207327.48 total 92.95 last 21.59) - open/close 1795 (time 1779207337.58 total 103.04 last 16.04) - open/close 1984 (time 1779207348.29 total 113.76 last 17.65) - open/close 2108 (time 1779207358.90 total 124.36 last 11.69) - open/close 2242 (time 1779207369.79 total 135.26 last 12.30) - open/close 2384 (time 1779207379.85 total 145.32 last 14.12) - open/close 2495 (time 1779207391.65 total 157.11 last 9.41) - open/close 2704 (time 1779207401.67 total 167.14 last 20.85) - open/close 2825 (time 1779207411.68 total 177.14 last 12.09) - open/close 3034 (time 1779207423.80 total 189.27 last 17.24) - open/close 3313 (time 1779207434.88 total 200.35 last 25.18) - open/close 3588 (time 1779207445.31 total 210.78 last 26.37) - open/close 3758 (time 1779207455.54 total 221.01 last 16.62) - open/close 3900 (time 1779207465.64 total 231.11 last 14.06) - open/close 4029 (time 1779207475.66 total 241.12 last 12.88) - open/close 4196 (time 1779207486.33 total 251.80 last 15.64) - open/close 4340 (time 1779207496.37 total 261.84 last 14.35) - open/close 4558 (time 1779207506.38 total 271.84 last 21.78) - open/close 4808 (time 1779207516.39 total 281.85 last 24.98) - open/close 4925 (time 1779207526.43 total 291.90 last 11.64) - open/close 5114 (time 1779207537.45 total 302.92 last 17.15) - open/close 5320 (time 1779207547.47 total 312.94 last 20.57) - open/close 5506 (time 1779207558.32 total 323.78 last 17.15) - open/close 5622 (time 1779207569.14 total 334.61 last 10.72) - open/close 5759 (time 1779207579.14 total 344.61 last 13.69) - open/close 5914 (time 1779207589.17 total 354.64 last 15.46) - open/close 6052 (time 1779207599.79 total 365.25 last 13.00) - open/close 6316 (time 1779207610.91 total 376.37 last 23.74) - open/close 6544 (time 1779207621.31 total 386.78 last 21.91) - open/close 6786 (time 1779207631.80 total 397.26 last 23.08) - open/close 7053 (time 1779207642.64 total 408.10 last 24.63) - open/close 7290 (time 1779207652.64 total 418.11 last 23.69) - open/close 7448 (time 1779207663.09 total 428.55 last 15.13) - open/close 7597 (time 1779207673.36 total 438.83 last 14.50) - open/close 7810 (time 1779207683.92 total 449.38 last 20.17) - open/close 8059 (time 1779207693.93 total 459.40 last 24.87) - open/close 8271 (time 1779207703.94 total 469.40 last 21.19) - open/close 8419 (time 1779207713.96 total 479.43 last 14.76) - open/close 8562 (time 1779207724.16 total 489.63 last 14.02) - open/close 8625 (time 1779207734.98 total 500.45 last 5.82) - open/close 8721 (time 1779207745.59 total 511.06 last 9.05) - open/close 8937 (time 1779207756.31 total 521.78 last 20.15) - open/close 9113 (time 1779207766.46 total 531.92 last 17.35) - open/close 9184 (time 1779207776.47 total 541.94 last 7.09) - open/close 9347 (time 1779207786.49 total 551.96 last 16.27) - open/close 9532 (time 1779207797.24 total 562.71 last 17.22) - open/close 9776 (time 1779207807.73 total 573.20 last 23.25) - open/close 10000 (time 1779207815.44 total 580.91 last 29.06) - open/close 10165 (time 1779207825.45 total 590.92 last 16.49) - open/close 10375 (time 1779207835.45 total 600.92 last 20.99) - open/close 10604 (time 1779207845.55 total 611.01 last 22.69) - open/close 10688 (time 1779207855.55 total 621.01 last 8.40) - open/close 10843 (time 1779207867.37 total 632.83 last 13.11) - open/close 11061 (time 1779207878.02 total 643.49 last 20.46) - open/close 11179 (time 1779207888.03 total 653.50 last 11.79) - open/close 11315 (time 1779207898.42 total 663.89 last 13.09) - open/close 11553 (time 1779207908.42 total 673.89 last 23.80) - open/close 11766 (time 1779207920.75 total 686.22 last 17.28) - open/close 12006 (time 1779207930.98 total 696.45 last 23.46) - open/close 12205 (time 1779207940.99 total 706.45 last 19.89) - open/close 12234 (time 1779207951.24 total 716.71 last 2.83) - open/close 12480 (time 1779207962.22 total 727.69 last 22.41) - open/close 12658 (time 1779207972.26 total 737.72 last 17.73) - open/close 12871 (time 1779207982.26 total 747.73 last 21.29) - open/close 13096 (time 1779207992.95 total 758.41 last 21.06) - open/close 13346 (time 1779208002.95 total 768.42 last 24.98) - open/close 13561 (time 1779208012.98 total 778.44 last 21.46) - open/close 13788 (time 1779208022.98 total 788.44 last 22.69) - open/close 13874 (time 1779208034.04 total 799.51 last 7.77) - open/close 13999 (time 1779208044.07 total 809.53 last 12.47) - open/close 14220 (time 1779208054.08 total 819.55 last 22.07) - open/close 14390 (time 1779208064.54 total 830.00 last 16.26) - open/close 14489 (time 1779208074.97 total 840.44 last 9.49) - open/close 14634 (time 1779208084.98 total 850.44 last 14.50) - open/close 14955 (time 1779208095.93 total 861.40 last 29.29) - open/close 15233 (time 1779208105.96 total 871.43 last 27.72) - open/close 15428 (time 1779208116.01 total 881.47 last 19.41) - open/close 15704 (time 1779208126.02 total 891.48 last 27.58) - open/close 15905 (time 1779208136.02 total 901.48 last 20.10) - open/close 16101 (time 1779208146.02 total 911.49 last 19.59) - open/close 16234 (time 1779208162.20 total 927.66 last 8.22) - open/close 16392 (time 1779208173.81 total 939.28 last 13.61) - open/close 16616 (time 1779208184.28 total 949.74 last 21.40) - open/close 16811 (time 1779208195.19 total 960.65 last 17.87) - open/close 16936 (time 1779208205.19 total 970.66 last 12.49) - open/close 17091 (time 1779208215.41 total 980.88 last 15.17) - open/close 17241 (time 1779208225.91 total 991.38 last 14.28) - open/close 17492 (time 1779208236.65 total 1002.11 last 23.38) - open/close 17832 (time 1779208246.66 total 1012.12 last 33.98) - open/close 18044 (time 1779208256.67 total 1022.14 last 21.16) - open/close 18355 (time 1779208267.25 total 1032.72 last 29.40) - open/close 18552 (time 1779208277.91 total 1043.37 last 18.49) - open/close 19318 (time 1779208287.91 total 1053.38 last 76.54) - open/close 20000 (time 1779208294.61 total 1060.08 last 101.81) - open/close 20943 (time 1779208304.61 total 1070.08 last 94.28) - open/close 21982 (time 1779208314.62 total 1080.09 last 103.80) - open/close 22882 (time 1779208324.63 total 1090.10 last 89.93) - open/close 23687 (time 1779208334.64 total 1100.10 last 80.45) - open/close 24557 (time 1779208344.64 total 1110.11 last 86.96) - open/close 25418 (time 1779208354.65 total 1120.12 last 86.03) - open/close 26400 (time 1779208364.66 total 1130.12 last 98.14) - open/close 27278 (time 1779208374.67 total 1140.13 last 87.71) - open/close 28111 (time 1779208384.67 total 1150.14 last 83.28) - open/close 29220 (time 1779208394.68 total 1160.14 last 110.83) - open/close 30000 (time 1779208404.06 total 1169.52 last 83.17) - open/close 30763 (time 1779208414.06 total 1179.52 last 76.27) - open/close 31554 (time 1779208424.06 total 1189.53 last 79.09) - open/close 32278 (time 1779208434.06 total 1199.53 last 72.38) - open/close 32990 (time 1779208444.06 total 1209.53 last 71.18) - open/close 33769 (time 1779208454.08 total 1219.54 last 77.81) - open/close 34568 (time 1779208464.08 total 1229.55 last 79.87) - open/close 35486 (time 1779208474.08 total 1239.55 last 91.77) - open/close 36380 (time 1779208484.09 total 1249.55 last 89.35) - open/close 37477 (time 1779208494.10 total 1259.57 last 109.56) - open/close 38323 (time 1779208504.11 total 1269.57 last 84.56) - open/close 39230 (time 1779208514.11 total 1279.57 last 90.68) - open/close 40000 (time 1779208522.90 total 1288.36 last 87.62) - open/close 40950 (time 1779208532.90 total 1298.37 last 94.95) - open/close 41815 (time 1779208542.90 total 1308.37 last 86.48) - open/close 42827 (time 1779208552.92 total 1318.38 last 101.08) - open/close 43678 (time 1779208562.92 total 1328.38 last 85.08) - open/close 44521 (time 1779208572.93 total 1338.39 last 84.22) - open/close 45412 (time 1779208582.93 total 1348.40 last 89.08) - open/close 46325 (time 1779208592.94 total 1358.40 last 91.23) - open/close 47355 (time 1779208602.94 total 1368.41 last 102.94) - open/close 48224 (time 1779208612.95 total 1378.41 last 86.85) - open/close 49141 (time 1779208622.95 total 1388.42 last 91.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1393.47 seconds: 35.62 ops/second - unlinked 0 (time 1779208632 ; total 0 ; last 0) - unlinked 10000 (time 1779208748 ; total 116 ; last 116) - unlinked 20000 (time 1779208861 ; total 229 ; last 113) - unlinked 30000 (time 1779208933 ; total 301 ; last 72) - unlinked 40000 (time 1779208999 ; total 367 ; last 66) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 424 seconds: 117.054245 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: modules unloaded. pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2