== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:26:06 (1779873966) start mds service on oleg214-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg214-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg214-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg214-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1779874052.87 total 10.08 last 21.42) - open/close 433 (time 1779874063.18 total 20.39 last 21.06) - open/close 663 (time 1779874073.29 total 30.51 last 22.73) - open/close 819 (time 1779874083.52 total 40.74 last 15.25) - open/close 982 (time 1779874093.65 total 50.86 last 16.10) - open/close 1175 (time 1779874103.66 total 60.87 last 19.28) - open/close 1386 (time 1779874114.51 total 71.72 last 19.44) - open/close 1502 (time 1779874125.49 total 82.71 last 10.56) - open/close 1644 (time 1779874135.51 total 92.72 last 14.17) - open/close 1795 (time 1779874146.51 total 103.73 last 13.73) - open/close 2044 (time 1779874157.20 total 114.41 last 23.30) - open/close 2167 (time 1779874167.82 total 125.04 last 11.58) - open/close 2403 (time 1779874178.32 total 135.53 last 22.49) - open/close 2523 (time 1779874188.82 total 146.03 last 11.43) - open/close 2781 (time 1779874199.60 total 156.82 last 23.91) - open/close 3005 (time 1779874209.67 total 166.88 last 22.26) - open/close 3168 (time 1779874219.92 total 177.14 last 15.89) - open/close 3351 (time 1779874229.94 total 187.15 last 18.27) - open/close 3565 (time 1779874240.91 total 198.12 last 19.52) - open/close 3819 (time 1779874250.92 total 208.13 last 25.36) - open/close 4058 (time 1779874260.94 total 218.15 last 23.85) - open/close 4300 (time 1779874272.30 total 229.51 last 21.31) - open/close 4554 (time 1779874282.50 total 239.71 last 24.90) - open/close 4801 (time 1779874292.51 total 249.72 last 24.68) - open/close 5043 (time 1779874303.12 total 260.33 last 22.80) - open/close 5361 (time 1779874313.18 total 270.39 last 31.62) - open/close 5614 (time 1779874323.39 total 280.61 last 24.76) - open/close 5902 (time 1779874333.40 total 290.61 last 28.78) - open/close 6009 (time 1779874343.40 total 300.62 last 10.70) - open/close 6191 (time 1779874354.40 total 311.61 last 16.56) - open/close 6329 (time 1779874364.40 total 321.61 last 13.79) - open/close 6579 (time 1779874374.91 total 332.13 last 23.78) - open/close 6682 (time 1779874387.71 total 344.92 last 8.05) - open/close 6948 (time 1779874397.99 total 355.20 last 25.88) - open/close 7206 (time 1779874408.00 total 365.21 last 25.78) - open/close 7467 (time 1779874418.00 total 375.21 last 26.09) - open/close 7724 (time 1779874428.31 total 385.53 last 24.92) - open/close 8004 (time 1779874439.03 total 396.24 last 26.13) - open/close 8220 (time 1779874449.47 total 406.69 last 20.68) - open/close 8340 (time 1779874460.51 total 417.72 last 10.88) - open/close 8495 (time 1779874470.52 total 427.73 last 15.48) - open/close 8732 (time 1779874480.52 total 437.74 last 23.69) - open/close 9066 (time 1779874490.77 total 447.98 last 32.61) - open/close 9265 (time 1779874500.79 total 458.00 last 19.86) - open/close 9536 (time 1779874510.80 total 468.01 last 27.08) - open/close 9846 (time 1779874521.25 total 478.46 last 29.67) - open/close 10000 (time 1779874526.87 total 484.08 last 27.37) - open/close 10235 (time 1779874537.16 total 494.38 last 22.83) - open/close 10466 (time 1779874547.17 total 504.38 last 23.09) - open/close 10647 (time 1779874559.24 total 516.45 last 14.99) - open/close 10949 (time 1779874570.09 total 527.30 last 27.84) - open/close 11171 (time 1779874581.05 total 538.26 last 20.25) - open/close 11368 (time 1779874591.27 total 548.49 last 19.27) - open/close 11676 (time 1779874601.97 total 559.18 last 28.80) - open/close 11976 (time 1779874611.97 total 569.19 last 29.99) - open/close 12249 (time 1779874622.02 total 579.23 last 27.18) - open/close 12489 (time 1779874632.58 total 589.79 last 22.72) - open/close 12765 (time 1779874642.65 total 599.86 last 27.42) - open/close 12955 (time 1779874653.07 total 610.29 last 18.22) - open/close 13188 (time 1779874663.95 total 621.16 last 21.42) - open/close 13489 (time 1779874673.96 total 631.17 last 30.08) - open/close 13748 (time 1779874684.31 total 641.52 last 25.01) - open/close 14019 (time 1779874694.32 total 651.54 last 27.07) - open/close 14166 (time 1779874705.24 total 662.46 last 13.46) - open/close 14282 (time 1779874715.46 total 672.68 last 11.35) - open/close 14614 (time 1779874726.31 total 683.53 last 30.60) - open/close 14827 (time 1779874736.35 total 693.56 last 21.22) - open/close 15037 (time 1779874746.65 total 703.86 last 20.40) - open/close 15313 (time 1779874756.68 total 713.89 last 27.52) - open/close 15472 (time 1779874766.69 total 723.91 last 15.87) - open/close 15750 (time 1779874777.29 total 734.51 last 26.23) - open/close 16010 (time 1779874787.33 total 744.54 last 25.90) - open/close 16221 (time 1779874797.36 total 754.57 last 21.04) - open/close 16365 (time 1779874807.37 total 764.58 last 14.39) - open/close 16630 (time 1779874817.57 total 774.78 last 25.97) - open/close 16923 (time 1779874828.32 total 785.53 last 27.27) - open/close 17186 (time 1779874838.61 total 795.82 last 25.55) - open/close 17468 (time 1779874848.62 total 805.84 last 28.17) - open/close 17675 (time 1779874859.32 total 816.53 last 19.35) - open/close 17969 (time 1779874870.28 total 827.49 last 26.84) - open/close 18139 (time 1779874880.28 total 837.49 last 16.99) - open/close 18318 (time 1779874890.78 total 847.99 last 17.05) - open/close 18532 (time 1779874900.78 total 858.00 last 21.39) - open/close 18667 (time 1779874910.80 total 868.01 last 13.48) - open/close 18910 (time 1779874921.43 total 878.64 last 22.87) - open/close 19196 (time 1779874932.36 total 889.57 last 26.16) - open/close 19350 (time 1779874942.36 total 899.57 last 15.39) - open/close 19550 (time 1779874953.03 total 910.24 last 18.74) - open/close 19796 (time 1779874963.54 total 920.75 last 23.41) - open/close 20000 (time 1779874973.15 total 930.36 last 21.23) - open/close 20267 (time 1779874983.17 total 940.38 last 26.66) - open/close 21202 (time 1779874993.18 total 950.39 last 93.40) - open/close 22356 (time 1779875003.18 total 960.39 last 115.35) - open/close 23367 (time 1779875013.18 total 970.40 last 101.06) - open/close 24416 (time 1779875023.19 total 980.40 last 104.89) - open/close 25460 (time 1779875033.19 total 990.40 last 104.34) - open/close 26468 (time 1779875043.20 total 1000.42 last 100.66) - open/close 27445 (time 1779875053.21 total 1010.42 last 97.66) - open/close 28298 (time 1779875063.21 total 1020.42 last 85.30) - open/close 29258 (time 1779875073.21 total 1030.43 last 95.96) - open/close 30000 (time 1779875082.05 total 1039.26 last 83.96) - open/close 30785 (time 1779875092.05 total 1049.26 last 78.49) - open/close 31764 (time 1779875102.06 total 1059.27 last 97.86) - open/close 32795 (time 1779875112.06 total 1069.28 last 103.02) - open/close 33781 (time 1779875122.07 total 1079.28 last 98.52) - open/close 34821 (time 1779875132.08 total 1089.29 last 103.91) - open/close 35784 (time 1779875142.09 total 1099.30 last 96.24) - open/close 36741 (time 1779875152.09 total 1109.30 last 95.64) - open/close 37766 (time 1779875162.10 total 1119.31 last 102.42) - open/close 38682 (time 1779875172.11 total 1129.32 last 91.52) - open/close 39519 (time 1779875182.12 total 1139.33 last 83.62) - open/close 40000 (time 1779875187.08 total 1144.30 last 96.86) - open/close 41034 (time 1779875197.09 total 1154.30 last 103.32) - open/close 41869 (time 1779875207.10 total 1164.31 last 83.43) - open/close 42932 (time 1779875217.11 total 1174.32 last 106.18) - open/close 43946 (time 1779875227.12 total 1184.33 last 101.36) - open/close 44967 (time 1779875237.12 total 1194.33 last 102.06) - open/close 45915 (time 1779875247.13 total 1204.34 last 94.74) - open/close 46793 (time 1779875257.13 total 1214.34 last 87.74) - open/close 47705 (time 1779875267.15 total 1224.36 last 91.07) - open/close 48420 (time 1779875277.15 total 1234.36 last 71.49) - open/close 49263 (time 1779875287.17 total 1244.38 last 84.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1248.27 seconds: 39.76 ops/second - unlinked 0 (time 1779875295 ; total 0 ; last 0) - unlinked 10000 (time 1779875399 ; total 104 ; last 104) - unlinked 20000 (time 1779875502 ; total 207 ; last 103) - unlinked 30000 (time 1779875564 ; total 269 ; last 62) - unlinked 40000 (time 1779875625 ; total 330 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 388 seconds: 127.914948 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local oleg214-server: modules unloaded. pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2