== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:54:57 (1775768097) start mds service on oleg658-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing load_modules_local oleg658-server: Loading modules from /home/green/git/lustre-release/lustre oleg658-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg658-server: detected 4 online CPUs by sysfs oleg658-server: MODOPTS_LIBCFS= oleg658-server: Force libcfs to create 2 CPU partitions oleg658-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg658-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg658-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg658-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg658-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-OST0000 oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg658-client.virtnet: -o user_xattr,flock 192.168.206.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1775768218.97 total 11.80 last 12.88) - open/close 304 (time 1775768229.02 total 21.85 last 15.12) - open/close 510 (time 1775768239.04 total 31.87 last 20.57) - open/close 671 (time 1775768249.56 total 42.39 last 15.31) - open/close 919 (time 1775768259.60 total 52.43 last 24.70) - open/close 1138 (time 1775768270.43 total 63.26 last 20.21) - open/close 1238 (time 1775768281.04 total 73.88 last 9.42) - open/close 1364 (time 1775768291.09 total 83.92 last 12.55) - open/close 1480 (time 1775768301.73 total 94.56 last 10.90) - open/close 1719 (time 1775768311.74 total 104.57 last 23.88) - open/close 1916 (time 1775768322.79 total 115.62 last 17.83) - open/close 2005 (time 1775768333.40 total 126.23 last 8.38) - open/close 2180 (time 1775768343.86 total 136.69 last 16.73) - open/close 2439 (time 1775768354.84 total 147.67 last 23.59) - open/close 2687 (time 1775768365.46 total 158.29 last 23.35) - open/close 2883 (time 1775768376.26 total 169.09 last 18.16) - open/close 3137 (time 1775768387.09 total 179.92 last 23.44) - open/close 3345 (time 1775768397.92 total 190.75 last 19.21) - open/close 3566 (time 1775768410.65 total 203.48 last 17.36) - open/close 3764 (time 1775768421.33 total 214.16 last 18.55) - open/close 3946 (time 1775768432.34 total 225.17 last 16.52) - open/close 4203 (time 1775768442.34 total 235.18 last 25.69) - open/close 4442 (time 1775768452.70 total 245.53 last 23.08) - open/close 4579 (time 1775768462.74 total 255.57 last 13.65) - open/close 4745 (time 1775768473.51 total 266.34 last 15.41) - open/close 4926 (time 1775768483.52 total 276.35 last 18.08) - open/close 5107 (time 1775768494.54 total 287.37 last 16.42) - open/close 5365 (time 1775768505.05 total 297.89 last 24.54) - open/close 5552 (time 1775768515.41 total 308.24 last 18.06) - open/close 5808 (time 1775768526.36 total 319.19 last 23.38) - open/close 6092 (time 1775768536.93 total 329.76 last 26.88) - open/close 6324 (time 1775768546.93 total 339.76 last 23.18) - open/close 6498 (time 1775768557.26 total 350.09 last 16.85) - open/close 6725 (time 1775768567.27 total 360.10 last 22.68) - open/close 6881 (time 1775768578.32 total 371.15 last 14.11) - open/close 7106 (time 1775768588.96 total 381.79 last 21.16) - open/close 7309 (time 1775768599.13 total 391.96 last 19.96) - open/close 7448 (time 1775768609.52 total 402.35 last 13.37) - open/close 7640 (time 1775768619.54 total 412.37 last 19.16) - open/close 7734 (time 1775768630.12 total 422.95 last 8.88) - open/close 7933 (time 1775768640.65 total 433.48 last 18.90) - open/close 8163 (time 1775768651.70 total 444.53 last 20.82) - open/close 8435 (time 1775768661.85 total 454.68 last 26.80) - open/close 8671 (time 1775768672.80 total 465.63 last 21.56) - open/close 8817 (time 1775768683.35 total 476.18 last 13.84) - open/close 8935 (time 1775768693.53 total 486.36 last 11.58) - open/close 9105 (time 1775768704.55 total 497.38 last 15.42) - open/close 9209 (time 1775768715.36 total 508.19 last 9.62) - open/close 9443 (time 1775768726.37 total 519.20 last 21.25) - open/close 9581 (time 1775768736.38 total 529.21 last 13.79) - open/close 9840 (time 1775768746.77 total 539.60 last 24.94) - open/close 9974 (time 1775768756.80 total 549.63 last 13.36) - open/close 10106 (time 1775768767.12 total 559.95 last 12.79) - open/close 10299 (time 1775768777.37 total 570.20 last 18.82) - open/close 10474 (time 1775768787.38 total 580.21 last 17.49) - open/close 10607 (time 1775768797.39 total 590.23 last 13.28) - open/close 10758 (time 1775768807.85 total 600.68 last 14.45) - open/close 10931 (time 1775768818.45 total 611.28 last 16.32) - open/close 11128 (time 1775768828.82 total 621.65 last 18.99) - open/close 11312 (time 1775768838.84 total 631.67 last 18.37) - open/close 11456 (time 1775768848.84 total 641.67 last 14.40) - open/close 11636 (time 1775768858.85 total 651.68 last 17.99) - open/close 11826 (time 1775768869.75 total 662.58 last 17.44) - open/close 11949 (time 1775768879.77 total 672.60 last 12.27) - open/close 12116 (time 1775768890.53 total 683.36 last 15.52) - open/close 12305 (time 1775768900.53 total 693.36 last 18.89) - open/close 12526 (time 1775768910.93 total 703.76 last 21.25) - open/close 12710 (time 1775768921.17 total 714.00 last 17.97) - open/close 12786 (time 1775768931.67 total 724.50 last 7.24) - open/close 12956 (time 1775768942.49 total 735.32 last 15.71) - open/close 13118 (time 1775768952.50 total 745.33 last 16.18) - open/close 13292 (time 1775768963.37 total 756.20 last 16.01) - open/close 13531 (time 1775768973.38 total 766.21 last 23.87) - open/close 13697 (time 1775768984.16 total 776.99 last 15.40) - open/close 13953 (time 1775768994.17 total 787.00 last 25.59) - open/close 14096 (time 1775769004.17 total 797.00 last 14.30) - open/close 14354 (time 1775769014.18 total 807.01 last 25.77) - open/close 14591 (time 1775769025.12 total 817.95 last 21.66) - open/close 14753 (time 1775769036.89 total 829.72 last 13.76) - open/close 14819 (time 1775769046.90 total 839.73 last 6.60) - open/close 14956 (time 1775769056.90 total 849.74 last 13.69) - open/close 15158 (time 1775769066.91 total 859.74 last 20.20) - open/close 15381 (time 1775769077.14 total 869.97 last 21.80) - open/close 15640 (time 1775769087.16 total 879.99 last 25.83) - open/close 15745 (time 1775769098.07 total 890.90 last 9.63) - open/close 15877 (time 1775769108.07 total 900.90 last 13.20) - open/close 16046 (time 1775769118.07 total 910.90 last 16.90) - open/close 16221 (time 1775769129.04 total 921.87 last 15.95) - open/close 16373 (time 1775769139.06 total 931.89 last 15.17) - open/close 16605 (time 1775769149.08 total 941.91 last 23.16) - open/close 16848 (time 1775769159.93 total 952.77 last 22.39) - open/close 17104 (time 1775769169.95 total 962.78 last 25.55) - open/close 17312 (time 1775769180.50 total 973.33 last 19.72) - open/close 17530 (time 1775769191.20 total 984.03 last 20.37) - open/close 17913 (time 1775769201.21 total 994.04 last 38.27) - open/close 18563 (time 1775769211.21 total 1004.04 last 64.98) - open/close 19212 (time 1775769221.22 total 1014.05 last 64.84) - open/close 19839 (time 1775769231.25 total 1024.08 last 62.50) - open/close 20000 (time 1775769233.92 total 1026.75 last 60.37) - open/close 20626 (time 1775769243.92 total 1036.75 last 62.59) - open/close 21230 (time 1775769253.93 total 1046.76 last 60.34) - open/close 21854 (time 1775769263.93 total 1056.76 last 62.40) - open/close 22454 (time 1775769273.94 total 1066.77 last 59.97) - open/close 23160 (time 1775769283.94 total 1076.77 last 70.54) - open/close 23850 (time 1775769293.95 total 1086.78 last 68.94) - open/close 24522 (time 1775769303.97 total 1096.80 last 67.11) - open/close 25183 (time 1775769313.97 total 1106.80 last 66.05) - open/close 25861 (time 1775769323.98 total 1116.81 last 67.73) - open/close 26536 (time 1775769333.98 total 1126.81 last 67.50) - open/close 27232 (time 1775769343.99 total 1136.82 last 69.56) - open/close 27888 (time 1775769354.00 total 1146.83 last 65.54) - open/close 28502 (time 1775769364.01 total 1156.84 last 61.33) - open/close 29224 (time 1775769374.02 total 1166.85 last 72.13) - open/close 29891 (time 1775769384.02 total 1176.85 last 66.69) - open/close 30569 (time 1775769394.02 total 1186.85 last 67.79) - open/close 31247 (time 1775769404.03 total 1196.86 last 67.72) - open/close 31964 (time 1775769414.03 total 1206.87 last 71.70) - open/close 32644 (time 1775769424.04 total 1216.87 last 67.99) - open/close 33349 (time 1775769434.04 total 1226.87 last 70.45) - open/close 34010 (time 1775769444.05 total 1236.88 last 66.04) - open/close 34656 (time 1775769454.06 total 1246.89 last 64.56) - open/close 35267 (time 1775769464.11 total 1256.94 last 60.82) - open/close 35922 (time 1775769474.12 total 1266.95 last 65.41) - open/close 36528 (time 1775769484.15 total 1276.98 last 60.43) - open/close 37248 (time 1775769494.17 total 1287.00 last 71.86) - open/close 37921 (time 1775769504.21 total 1297.05 last 66.97) - open/close 38626 (time 1775769514.23 total 1307.06 last 70.40) - open/close 39340 (time 1775769524.25 total 1317.08 last 71.26) - open/close 39986 (time 1775769534.26 total 1327.09 last 64.54) - open/close 40664 (time 1775769544.27 total 1337.10 last 67.74) - open/close 41382 (time 1775769554.27 total 1347.10 last 71.75) - open/close 42119 (time 1775769564.28 total 1357.11 last 73.65) - open/close 42827 (time 1775769574.28 total 1367.11 last 70.78) - open/close 43548 (time 1775769584.29 total 1377.12 last 72.07) - open/close 44277 (time 1775769594.29 total 1387.12 last 72.86) - open/close 44938 (time 1775769604.30 total 1397.13 last 66.05) - open/close 45637 (time 1775769614.32 total 1407.15 last 69.76) - open/close 46296 (time 1775769624.33 total 1417.16 last 65.85) - open/close 46959 (time 1775769634.37 total 1427.20 last 66.00) - open/close 47667 (time 1775769644.39 total 1437.22 last 70.70) - open/close 48333 (time 1775769654.41 total 1447.24 last 66.42) - open/close 49011 (time 1775769664.43 total 1457.26 last 67.72) - open/close 49601 (time 1775769674.43 total 1467.26 last 59.00) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1467.72 seconds: 33.82 ops/second - unlinked 0 (time 1775769678 ; total 0 ; last 0) - unlinked 10000 (time 1775769840 ; total 162 ; last 162) - unlinked 20000 (time 1775770004 ; total 326 ; last 164) - unlinked 30000 (time 1775770166 ; total 488 ; last 162) - unlinked 40000 (time 1775770330 ; total 652 ; last 164) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 807 seconds: 61.501858 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg658-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg658-server unloading modules via unload_modules_local on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing unload_modules_local oleg658-server: modules unloaded. pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2