== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:34:36 (1787384076) start mds service on oleg645-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg645-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg645-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1787384157.26 total 10.01 last 25.77) - open/close 398 (time 1787384167.49 total 20.24 last 13.68) - open/close 649 (time 1787384178.30 total 31.05 last 23.23) - open/close 910 (time 1787384189.05 total 41.80 last 24.28) - open/close 1080 (time 1787384199.54 total 52.30 last 16.20) - open/close 1331 (time 1787384210.45 total 63.20 last 23.01) - open/close 1459 (time 1787384221.52 total 74.27 last 11.57) - open/close 1577 (time 1787384231.53 total 84.28 last 11.79) - open/close 1806 (time 1787384244.23 total 96.98 last 18.03) - open/close 2069 (time 1787384255.03 total 107.78 last 24.35) - open/close 2339 (time 1787384265.59 total 118.35 last 25.56) - open/close 2634 (time 1787384275.61 total 128.36 last 29.45) - open/close 2908 (time 1787384285.62 total 138.38 last 27.37) - open/close 3124 (time 1787384295.63 total 148.39 last 21.57) - open/close 3362 (time 1787384305.65 total 158.40 last 23.77) - open/close 3590 (time 1787384315.86 total 168.61 last 22.32) - open/close 3858 (time 1787384326.64 total 179.40 last 24.85) - open/close 4160 (time 1787384337.52 total 190.27 last 27.77) - open/close 4408 (time 1787384347.54 total 200.29 last 24.75) - open/close 4597 (time 1787384358.06 total 210.81 last 17.96) - open/close 4824 (time 1787384368.06 total 220.82 last 22.69) - open/close 5070 (time 1787384378.37 total 231.12 last 23.87) - open/close 5300 (time 1787384388.41 total 241.16 last 22.91) - open/close 5526 (time 1787384398.43 total 251.18 last 22.55) - open/close 5735 (time 1787384408.56 total 261.32 last 20.62) - open/close 5950 (time 1787384418.71 total 271.46 last 21.19) - open/close 6185 (time 1787384428.75 total 281.51 last 23.40) - open/close 6374 (time 1787384439.24 total 291.99 last 18.02) - open/close 6599 (time 1787384449.25 total 302.01 last 22.47) - open/close 6812 (time 1787384459.58 total 312.33 last 20.63) - open/close 7119 (time 1787384470.32 total 323.07 last 28.58) - open/close 7402 (time 1787384480.66 total 333.41 last 27.38) - open/close 7601 (time 1787384490.90 total 343.65 last 19.43) - open/close 7791 (time 1787384500.90 total 353.65 last 19.00) - open/close 8063 (time 1787384511.80 total 364.56 last 24.94) - open/close 8290 (time 1787384521.81 total 374.57 last 22.68) - open/close 8570 (time 1787384532.53 total 385.29 last 26.12) - open/close 8796 (time 1787384542.54 total 395.30 last 22.57) - open/close 9001 (time 1787384553.14 total 405.90 last 19.34) - open/close 9274 (time 1787384564.14 total 416.90 last 24.82) - open/close 9499 (time 1787384574.96 total 427.71 last 20.81) - open/close 9740 (time 1787384585.34 total 438.09 last 23.22) - open/close 10000 (time 1787384593.41 total 446.16 last 32.22) - open/close 10228 (time 1787384603.45 total 456.20 last 22.71) - open/close 10402 (time 1787384613.49 total 466.24 last 17.34) - open/close 10572 (time 1787384624.18 total 476.94 last 15.89) - open/close 10762 (time 1787384634.37 total 487.12 last 18.65) - open/close 10964 (time 1787384645.10 total 497.86 last 18.82) - open/close 11156 (time 1787384655.14 total 507.89 last 19.14) - open/close 11380 (time 1787384665.55 total 518.31 last 21.51) - open/close 11617 (time 1787384676.04 total 528.79 last 22.60) - open/close 11825 (time 1787384686.06 total 538.82 last 20.74) - open/close 12075 (time 1787384696.84 total 549.59 last 23.21) - open/close 12355 (time 1787384706.84 total 559.60 last 27.99) - open/close 12580 (time 1787384716.86 total 569.61 last 22.46) - open/close 12737 (time 1787384727.04 total 579.80 last 15.42) - open/close 12939 (time 1787384737.05 total 589.81 last 20.18) - open/close 13182 (time 1787384747.83 total 600.59 last 22.54) - open/close 13400 (time 1787384758.32 total 611.07 last 20.79) - open/close 13626 (time 1787384768.33 total 621.08 last 22.58) - open/close 13858 (time 1787384778.34 total 631.10 last 23.16) - open/close 14049 (time 1787384788.47 total 641.23 last 18.85) - open/close 14265 (time 1787384799.23 total 651.98 last 20.08) - open/close 14494 (time 1787384809.24 total 662.00 last 22.87) - open/close 14711 (time 1787384819.25 total 672.01 last 21.69) - open/close 14989 (time 1787384829.26 total 682.02 last 27.77) - open/close 15157 (time 1787384839.99 total 692.75 last 15.66) - open/close 15412 (time 1787384850.50 total 703.26 last 24.26) - open/close 15638 (time 1787384860.51 total 713.26 last 22.59) - open/close 15743 (time 1787384871.54 total 724.29 last 9.52) - open/close 16008 (time 1787384882.29 total 735.04 last 24.64) - open/close 16276 (time 1787384892.48 total 745.24 last 26.29) - open/close 16538 (time 1787384903.56 total 756.31 last 23.66) - open/close 16808 (time 1787384913.74 total 766.49 last 26.53) - open/close 17024 (time 1787384924.35 total 777.10 last 20.35) - open/close 17265 (time 1787384934.36 total 787.12 last 24.07) - open/close 17470 (time 1787384944.78 total 797.53 last 19.68) - open/close 17667 (time 1787384955.79 total 808.54 last 17.89) - open/close 17868 (time 1787384965.80 total 818.56 last 20.07) - open/close 18072 (time 1787384975.82 total 828.58 last 20.36) - open/close 18348 (time 1787384986.90 total 839.65 last 24.92) - open/close 18655 (time 1787384996.90 total 849.65 last 30.70) - open/close 18865 (time 1787385007.66 total 860.42 last 19.51) - open/close 19121 (time 1787385017.67 total 870.42 last 25.59) - open/close 19607 (time 1787385027.67 total 880.42 last 48.60) - open/close 20000 (time 1787385034.68 total 887.44 last 56.04) - open/close 20676 (time 1787385044.69 total 897.45 last 67.52) - open/close 21333 (time 1787385054.69 total 907.45 last 65.70) - open/close 22085 (time 1787385064.69 total 917.45 last 75.20) - open/close 22772 (time 1787385074.70 total 927.45 last 68.66) - open/close 23429 (time 1787385084.70 total 937.46 last 65.67) - open/close 24100 (time 1787385094.71 total 947.46 last 67.06) - open/close 24784 (time 1787385104.72 total 957.47 last 68.34) - open/close 25376 (time 1787385114.74 total 967.49 last 59.08) - open/close 25980 (time 1787385124.75 total 977.51 last 60.32) - open/close 26618 (time 1787385134.75 total 987.51 last 63.78) - open/close 27257 (time 1787385144.77 total 997.53 last 63.77) - open/close 27905 (time 1787385154.78 total 1007.53 last 64.78) - open/close 28539 (time 1787385164.78 total 1017.54 last 63.35) - open/close 29216 (time 1787385174.80 total 1027.56 last 67.58) - open/close 29840 (time 1787385184.81 total 1037.56 last 62.37) - open/close 30000 (time 1787385187.84 total 1040.59 last 52.79) - open/close 30639 (time 1787385197.85 total 1050.60 last 63.85) - open/close 31308 (time 1787385207.86 total 1060.62 last 66.82) - open/close 32008 (time 1787385217.86 total 1070.62 last 69.98) - open/close 32752 (time 1787385227.86 total 1080.62 last 74.39) - open/close 33443 (time 1787385237.87 total 1090.62 last 69.08) - open/close 34088 (time 1787385247.87 total 1100.62 last 64.50) - open/close 34770 (time 1787385257.87 total 1110.62 last 68.20) - open/close 35501 (time 1787385267.89 total 1120.64 last 72.97) - open/close 36220 (time 1787385277.89 total 1130.64 last 71.88) - open/close 36843 (time 1787385287.89 total 1140.65 last 62.28) - open/close 37600 (time 1787385297.89 total 1150.65 last 75.69) - open/close 38405 (time 1787385307.90 total 1160.65 last 80.45) - open/close 39084 (time 1787385317.91 total 1170.66 last 67.83) - open/close 39679 (time 1787385327.91 total 1180.67 last 59.50) - open/close 40000 (time 1787385332.32 total 1185.08 last 72.73) - open/close 40684 (time 1787385342.33 total 1195.09 last 68.34) - open/close 41367 (time 1787385352.33 total 1205.09 last 68.28) - open/close 42052 (time 1787385362.34 total 1215.09 last 68.47) - open/close 42791 (time 1787385372.34 total 1225.10 last 73.86) - open/close 43532 (time 1787385382.34 total 1235.10 last 74.09) - open/close 44257 (time 1787385392.35 total 1245.11 last 72.43) - open/close 44897 (time 1787385402.36 total 1255.12 last 63.94) - open/close 45553 (time 1787385412.37 total 1265.12 last 65.57) - open/close 46229 (time 1787385422.37 total 1275.13 last 67.55) - open/close 46925 (time 1787385432.39 total 1285.14 last 69.49) - open/close 47616 (time 1787385442.39 total 1295.15 last 69.08) - open/close 48330 (time 1787385452.41 total 1305.17 last 71.26) - open/close 49006 (time 1787385462.42 total 1315.18 last 67.51) - open/close 49662 (time 1787385472.43 total 1325.18 last 65.58) total: 50000 open/close in 1329.98 seconds: 37.59 ops/second - unlinked 0 (time 1787385481 ; total 0 ; last 0) - unlinked 10000 (time 1787385640 ; total 159 ; last 159) - unlinked 20000 (time 1787385796 ; total 315 ; last 156) - unlinked 30000 (time 1787385963 ; total 482 ; last 167) - unlinked 40000 (time 1787386131 ; total 650 ; last 168) total: 50000 unlinks in 817 seconds: 61.199509 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: modules unloaded. pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2