== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:37:20 (1779993440) start mds service on oleg206-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg206-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg206-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 172 (time 1779993528.21 total 10.87 last 15.82) - open/close 369 (time 1779993538.58 total 21.24 last 19.01) - open/close 620 (time 1779993549.54 total 32.20 last 22.90) - open/close 793 (time 1779993559.54 total 42.20 last 17.29) - open/close 1010 (time 1779993570.40 total 53.06 last 19.99) - open/close 1064 (time 1779993581.22 total 63.88 last 4.99) - open/close 1200 (time 1779993591.23 total 73.89 last 13.58) - open/close 1445 (time 1779993603.41 total 86.07 last 20.12) - open/close 1634 (time 1779993613.90 total 96.56 last 18.01) - open/close 1755 (time 1779993624.21 total 106.87 last 11.74) - open/close 1786 (time 1779993634.46 total 117.12 last 3.03) - open/close 2005 (time 1779993644.46 total 127.12 last 21.88) - open/close 2122 (time 1779993655.51 total 138.17 last 10.59) - open/close 2308 (time 1779993665.84 total 148.50 last 18.01) - open/close 2477 (time 1779993676.62 total 159.28 last 15.68) - open/close 2664 (time 1779993686.63 total 169.29 last 18.68) - open/close 2820 (time 1779993696.63 total 179.29 last 15.59) - open/close 3002 (time 1779993706.74 total 189.40 last 18.01) - open/close 3074 (time 1779993716.82 total 199.48 last 7.14) - open/close 3107 (time 1779993727.18 total 209.84 last 3.18) - open/close 3308 (time 1779993738.14 total 220.80 last 18.34) - open/close 3545 (time 1779993748.99 total 231.65 last 21.85) - open/close 3776 (time 1779993759.01 total 241.67 last 23.06) - open/close 3872 (time 1779993769.34 total 252.00 last 9.29) - open/close 4066 (time 1779993779.34 total 262.00 last 19.39) - open/close 4293 (time 1779993789.36 total 272.02 last 22.66) - open/close 4406 (time 1779993801.18 total 283.84 last 9.55) - open/close 4534 (time 1779993811.34 total 294.00 last 12.61) - open/close 4690 (time 1779993821.38 total 304.04 last 15.53) - open/close 4838 (time 1779993832.09 total 314.75 last 13.82) - open/close 5015 (time 1779993843.11 total 325.77 last 16.07) - open/close 5263 (time 1779993853.11 total 335.77 last 24.79) - open/close 5541 (time 1779993864.09 total 346.75 last 25.32) - open/close 5786 (time 1779993874.10 total 356.76 last 24.47) - open/close 6017 (time 1779993884.68 total 367.34 last 21.83) - open/close 6283 (time 1779993895.53 total 378.19 last 24.53) - open/close 6438 (time 1779993906.26 total 388.92 last 14.44) - open/close 6644 (time 1779993917.02 total 399.68 last 19.15) - open/close 6767 (time 1779993927.04 total 409.70 last 12.28) - open/close 7008 (time 1779993937.36 total 420.02 last 23.34) - open/close 7196 (time 1779993947.37 total 430.03 last 18.79) - open/close 7355 (time 1779993957.37 total 440.03 last 15.90) - open/close 7588 (time 1779993968.66 total 451.32 last 20.64) - open/close 7743 (time 1779993979.46 total 462.12 last 14.34) - open/close 7815 (time 1779993990.17 total 472.83 last 6.72) - open/close 7900 (time 1779994000.46 total 483.12 last 8.27) - open/close 8098 (time 1779994010.89 total 493.55 last 18.98) - open/close 8371 (time 1779994021.72 total 504.38 last 25.20) - open/close 8652 (time 1779994031.73 total 514.39 last 28.08) - open/close 8810 (time 1779994041.76 total 524.42 last 15.75) - open/close 8898 (time 1779994052.24 total 534.90 last 8.39) - open/close 9018 (time 1779994062.26 total 544.92 last 11.98) - open/close 9151 (time 1779994072.40 total 555.06 last 13.11) - open/close 9284 (time 1779994083.37 total 566.03 last 12.13) - open/close 9437 (time 1779994093.93 total 576.59 last 14.50) - open/close 9616 (time 1779994105.38 total 588.04 last 15.63) - open/close 9763 (time 1779994117.52 total 600.18 last 12.11) - open/close 9934 (time 1779994128.59 total 611.25 last 15.45) - open/close 10000 (time 1779994130.80 total 613.46 last 29.79) - open/close 10228 (time 1779994140.80 total 623.46 last 22.79) - open/close 10490 (time 1779994151.08 total 633.74 last 25.51) - open/close 10631 (time 1779994161.81 total 644.47 last 13.14) - open/close 10852 (time 1779994172.15 total 654.81 last 21.37) - open/close 11013 (time 1779994182.15 total 664.81 last 16.10) - open/close 11211 (time 1779994192.86 total 675.52 last 18.48) - open/close 11378 (time 1779994203.54 total 686.20 last 15.64) - open/close 11559 (time 1779994213.54 total 696.20 last 18.10) - open/close 11792 (time 1779994224.09 total 706.75 last 22.09) - open/close 11874 (time 1779994239.87 total 722.53 last 5.20) - open/close 11968 (time 1779994250.65 total 733.31 last 8.72) - open/close 12097 (time 1779994261.64 total 744.30 last 11.73) - open/close 12144 (time 1779994272.33 total 754.99 last 4.40) - open/close 12292 (time 1779994282.49 total 765.15 last 14.56) - open/close 12461 (time 1779994293.04 total 775.70 last 16.02) - open/close 12551 (time 1779994305.84 total 788.50 last 7.03) - open/close 12727 (time 1779994315.94 total 798.60 last 17.43) - open/close 12945 (time 1779994326.48 total 809.14 last 20.68) - open/close 13142 (time 1779994337.23 total 819.89 last 18.31) - open/close 13253 (time 1779994347.49 total 830.15 last 10.82) - open/close 13437 (time 1779994357.71 total 840.37 last 18.01) - open/close 13617 (time 1779994367.74 total 850.40 last 17.96) - open/close 13710 (time 1779994378.70 total 861.36 last 8.48) - open/close 13886 (time 1779994388.71 total 871.37 last 17.59) - open/close 14044 (time 1779994399.31 total 881.97 last 14.90) - open/close 14098 (time 1779994410.11 total 892.77 last 5.00) - open/close 14267 (time 1779994420.53 total 903.19 last 16.23) - open/close 14454 (time 1779994430.53 total 913.19 last 18.69) - open/close 14565 (time 1779994441.40 total 924.06 last 10.21) - open/close 14646 (time 1779994451.73 total 934.39 last 7.84) - open/close 14868 (time 1779994462.43 total 945.09 last 20.74) - open/close 14917 (time 1779994474.19 total 956.85 last 4.17) - open/close 14963 (time 1779994484.78 total 967.44 last 4.35) - open/close 15149 (time 1779994494.78 total 977.44 last 18.59) - open/close 15221 (time 1779994504.78 total 987.44 last 7.20) - open/close 15365 (time 1779994515.75 total 998.41 last 13.13) - open/close 15560 (time 1779994525.92 total 1008.58 last 19.19) - open/close 15733 (time 1779994535.92 total 1018.58 last 17.29) - open/close 15979 (time 1779994546.26 total 1028.92 last 23.80) - open/close 16139 (time 1779994556.84 total 1039.50 last 15.12) - open/close 16325 (time 1779994567.84 total 1050.50 last 16.91) - open/close 16441 (time 1779994577.84 total 1060.50 last 11.59) - open/close 16963 (time 1779994587.85 total 1070.51 last 52.19) - open/close 17625 (time 1779994597.86 total 1080.52 last 66.14) - open/close 18336 (time 1779994607.86 total 1090.52 last 71.05) - open/close 19066 (time 1779994617.87 total 1100.53 last 72.92) - open/close 19651 (time 1779994627.87 total 1110.53 last 58.49) - open/close 20000 (time 1779994632.77 total 1115.43 last 71.32) - open/close 20788 (time 1779994642.77 total 1125.43 last 78.78) - open/close 21472 (time 1779994652.77 total 1135.43 last 68.39) - open/close 22097 (time 1779994662.77 total 1145.43 last 62.50) - open/close 22779 (time 1779994672.78 total 1155.44 last 68.14) - open/close 23387 (time 1779994682.78 total 1165.44 last 60.80) - open/close 23888 (time 1779994692.78 total 1175.44 last 50.09) - open/close 24505 (time 1779994702.78 total 1185.44 last 61.70) - open/close 25167 (time 1779994712.78 total 1195.44 last 66.20) - open/close 25886 (time 1779994722.79 total 1205.45 last 71.86) - open/close 26659 (time 1779994732.80 total 1215.46 last 77.21) - open/close 27321 (time 1779994742.82 total 1225.48 last 66.08) - open/close 28095 (time 1779994752.83 total 1235.49 last 77.34) - open/close 28817 (time 1779994762.84 total 1245.50 last 72.12) - open/close 29646 (time 1779994772.84 total 1255.50 last 82.88) - open/close 30000 (time 1779994777.83 total 1260.49 last 71.00) - open/close 30596 (time 1779994787.83 total 1270.49 last 59.57) - open/close 31164 (time 1779994797.84 total 1280.50 last 56.77) - open/close 31853 (time 1779994807.85 total 1290.51 last 68.84) - open/close 32594 (time 1779994817.86 total 1300.52 last 73.99) - open/close 33325 (time 1779994827.87 total 1310.53 last 73.07) - open/close 34122 (time 1779994837.87 total 1320.53 last 79.70) - open/close 34922 (time 1779994847.87 total 1330.53 last 79.93) - open/close 35627 (time 1779994857.88 total 1340.54 last 70.48) - open/close 36177 (time 1779994867.89 total 1350.55 last 54.95) - open/close 36816 (time 1779994877.90 total 1360.56 last 63.82) - open/close 37386 (time 1779994887.92 total 1370.58 last 56.88) - open/close 38067 (time 1779994897.94 total 1380.60 last 67.99) - open/close 38674 (time 1779994907.94 total 1390.60 last 60.67) - open/close 39410 (time 1779994917.96 total 1400.62 last 73.46) - open/close 40000 (time 1779994927.02 total 1409.68 last 65.14) - open/close 40681 (time 1779994937.03 total 1419.69 last 68.04) - open/close 41386 (time 1779994947.03 total 1429.69 last 70.49) - open/close 42024 (time 1779994957.05 total 1439.71 last 63.68) - open/close 42663 (time 1779994967.06 total 1449.72 last 63.80) - open/close 43385 (time 1779994977.06 total 1459.72 last 72.19) - open/close 44234 (time 1779994987.07 total 1469.73 last 84.88) - open/close 45012 (time 1779994997.09 total 1479.75 last 77.64) - open/close 45633 (time 1779995007.10 total 1489.76 last 62.03) - open/close 46245 (time 1779995017.12 total 1499.78 last 61.10) - open/close 46926 (time 1779995027.12 total 1509.78 last 68.06) - open/close 47717 (time 1779995037.12 total 1519.78 last 79.09) - open/close 48529 (time 1779995047.13 total 1529.79 last 81.18) - open/close 49338 (time 1779995057.13 total 1539.79 last 80.85) total: 50000 open/close in 1548.65 seconds: 32.29 ops/second - unlinked 0 (time 1779995070 ; total 0 ; last 0) - unlinked 10000 (time 1779995210 ; total 140 ; last 140) - unlinked 20000 (time 1779995349 ; total 279 ; last 139) - unlinked 30000 (time 1779995495 ; total 425 ; last 146) - unlinked 40000 (time 1779995633 ; total 563 ; last 138) total: 50000 unlinks in 707 seconds: 70.721359 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2