== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:27:43 (1787790463) start mds service on oleg256-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg256-server: detected 4 online CPUs by sysfs oleg256-server: MODOPTS_LIBCFS= oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg256-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 222 (time 1787790575.94 total 10.01 last 22.18) - open/close 488 (time 1787790585.95 total 20.02 last 26.56) - open/close 700 (time 1787790596.59 total 30.66 last 19.93) - open/close 963 (time 1787790607.16 total 41.23 last 24.89) - open/close 1216 (time 1787790618.03 total 52.11 last 23.26) - open/close 1359 (time 1787790629.05 total 63.12 last 12.98) - open/close 1554 (time 1787790639.09 total 73.16 last 19.42) - open/close 1830 (time 1787790649.09 total 83.16 last 27.60) - open/close 2019 (time 1787790659.26 total 93.33 last 18.59) - open/close 2260 (time 1787790669.28 total 103.35 last 24.05) - open/close 2542 (time 1787790679.68 total 113.75 last 27.11) - open/close 2838 (time 1787790690.29 total 124.36 last 27.89) - open/close 3099 (time 1787790700.68 total 134.75 last 25.12) - open/close 3347 (time 1787790710.90 total 144.97 last 24.26) - open/close 3568 (time 1787790721.23 total 155.30 last 21.40) - open/close 3838 (time 1787790731.25 total 165.32 last 26.94) - open/close 4056 (time 1787790741.57 total 175.64 last 21.14) - open/close 4309 (time 1787790751.58 total 185.65 last 25.27) - open/close 4539 (time 1787790761.97 total 196.04 last 22.13) - open/close 4761 (time 1787790771.98 total 206.05 last 22.18) - open/close 5030 (time 1787790782.20 total 216.27 last 26.32) - open/close 5267 (time 1787790792.23 total 226.30 last 23.64) - open/close 5513 (time 1787790802.23 total 236.31 last 24.58) - open/close 5730 (time 1787790812.30 total 246.37 last 21.55) - open/close 5859 (time 1787790822.31 total 256.38 last 12.89) - open/close 6030 (time 1787790832.77 total 266.84 last 16.34) - open/close 6277 (time 1787790843.65 total 277.73 last 22.70) - open/close 6524 (time 1787790854.32 total 288.39 last 23.16) - open/close 6774 (time 1787790864.85 total 298.92 last 23.73) - open/close 7055 (time 1787790875.88 total 309.96 last 25.47) - open/close 7232 (time 1787790886.06 total 320.13 last 17.39) - open/close 7392 (time 1787790896.07 total 330.14 last 15.99) - open/close 7711 (time 1787790906.08 total 340.15 last 31.88) - open/close 8002 (time 1787790916.21 total 350.28 last 28.71) - open/close 8257 (time 1787790926.44 total 360.51 last 24.93) - open/close 8518 (time 1787790937.01 total 371.08 last 24.71) - open/close 8818 (time 1787790947.47 total 381.55 last 28.66) - open/close 9084 (time 1787790957.50 total 391.57 last 26.55) - open/close 9266 (time 1787790968.58 total 402.65 last 16.42) - open/close 9417 (time 1787790979.14 total 413.21 last 14.30) - open/close 9648 (time 1787790989.71 total 423.78 last 21.85) - open/close 9848 (time 1787790999.93 total 434.00 last 19.57) - open/close 9992 (time 1787791010.82 total 444.89 last 13.23) - open/close 10315 (time 1787791021.50 total 455.57 last 30.25) - open/close 10521 (time 1787791031.51 total 465.58 last 20.56) - open/close 10791 (time 1787791042.16 total 476.23 last 25.35) - open/close 11036 (time 1787791053.14 total 487.21 last 22.33) - open/close 11210 (time 1787791063.53 total 497.60 last 16.74) - open/close 11369 (time 1787791073.77 total 507.84 last 15.53) - open/close 11591 (time 1787791084.02 total 518.09 last 21.65) - open/close 11834 (time 1787791094.47 total 528.54 last 23.26) - open/close 12134 (time 1787791104.57 total 538.64 last 29.70) - open/close 12385 (time 1787791114.80 total 548.87 last 24.53) - open/close 12648 (time 1787791125.75 total 559.82 last 24.03) - open/close 12875 (time 1787791135.75 total 569.82 last 22.70) - open/close 13076 (time 1787791145.76 total 579.83 last 20.07) - open/close 13244 (time 1787791155.95 total 590.02 last 16.49) - open/close 13498 (time 1787791168.10 total 602.17 last 20.91) - open/close 13664 (time 1787791178.61 total 612.68 last 15.79) - open/close 13936 (time 1787791188.64 total 622.71 last 27.12) - open/close 14161 (time 1787791198.65 total 632.72 last 22.48) - open/close 14429 (time 1787791209.33 total 643.40 last 25.09) - open/close 14550 (time 1787791219.89 total 653.97 last 11.45) - open/close 14684 (time 1787791230.14 total 664.21 last 13.08) - open/close 14922 (time 1787791240.88 total 674.95 last 22.16) - open/close 15126 (time 1787791251.97 total 686.04 last 18.40) - open/close 15399 (time 1787791261.97 total 696.05 last 27.28) - open/close 15595 (time 1787791271.98 total 706.05 last 19.59) - open/close 15896 (time 1787791282.74 total 716.81 last 27.99) - open/close 16098 (time 1787791293.11 total 727.18 last 19.47) - open/close 16277 (time 1787791303.13 total 737.20 last 17.86) - open/close 16499 (time 1787791313.46 total 747.53 last 21.50) - open/close 16747 (time 1787791323.47 total 757.54 last 24.78) - open/close 17014 (time 1787791334.21 total 768.28 last 24.86) - open/close 17179 (time 1787791344.22 total 778.29 last 16.47) - open/close 17350 (time 1787791354.44 total 788.51 last 16.74) - open/close 17579 (time 1787791364.92 total 798.99 last 21.86) - open/close 17840 (time 1787791375.68 total 809.75 last 24.25) - open/close 18123 (time 1787791385.69 total 819.76 last 28.27) - open/close 18370 (time 1787791396.31 total 830.38 last 23.25) - open/close 18628 (time 1787791407.18 total 841.26 last 23.73) - open/close 18873 (time 1787791417.20 total 851.27 last 24.46) - open/close 19086 (time 1787791427.53 total 861.60 last 20.62) - open/close 19183 (time 1787791437.54 total 871.61 last 9.69) - open/close 19405 (time 1787791447.55 total 881.62 last 22.19) - open/close 19642 (time 1787791457.86 total 891.93 last 22.99) - open/close 19803 (time 1787791467.86 total 901.93 last 16.10) - open/close 19990 (time 1787791477.87 total 911.94 last 18.69) - open/close 20158 (time 1787791488.71 total 922.78 last 15.49) - open/close 20953 (time 1787791498.72 total 932.79 last 79.44) - open/close 21896 (time 1787791508.72 total 942.79 last 94.28) - open/close 22733 (time 1787791518.72 total 952.79 last 83.68) - open/close 23535 (time 1787791528.73 total 962.80 last 80.14) - open/close 24286 (time 1787791538.74 total 972.81 last 75.04) - open/close 25233 (time 1787791548.74 total 982.81 last 94.63) - open/close 26104 (time 1787791558.75 total 992.82 last 87.04) - open/close 27026 (time 1787791568.76 total 1002.83 last 92.12) - open/close 27950 (time 1787791578.77 total 1012.84 last 92.33) - open/close 28920 (time 1787791588.77 total 1022.84 last 96.99) - open/close 29794 (time 1787791598.78 total 1032.85 last 87.32) - open/close 30000 (time 1787791601.78 total 1035.85 last 68.68) - open/close 30952 (time 1787791611.78 total 1045.85 last 95.16) - open/close 31797 (time 1787791621.79 total 1055.86 last 84.44) - open/close 32702 (time 1787791631.79 total 1065.86 last 90.49) - open/close 33498 (time 1787791641.79 total 1075.86 last 79.59) - open/close 34432 (time 1787791651.79 total 1085.86 last 93.38) - open/close 35256 (time 1787791661.79 total 1095.86 last 82.39) - open/close 36116 (time 1787791671.80 total 1105.87 last 85.97) - open/close 37032 (time 1787791681.80 total 1115.87 last 91.59) - open/close 37861 (time 1787791691.80 total 1125.88 last 82.84) - open/close 38607 (time 1787791701.81 total 1135.88 last 74.59) - open/close 39372 (time 1787791711.81 total 1145.88 last 76.44) - open/close 40000 (time 1787791719.03 total 1153.10 last 87.05) - open/close 40925 (time 1787791729.05 total 1163.12 last 92.34) - open/close 41908 (time 1787791739.06 total 1173.13 last 98.21) - open/close 42801 (time 1787791749.06 total 1183.13 last 89.27) - open/close 43612 (time 1787791759.07 total 1193.14 last 81.04) - open/close 44422 (time 1787791769.07 total 1203.14 last 80.95) - open/close 45302 (time 1787791779.08 total 1213.15 last 87.91) - open/close 46252 (time 1787791789.08 total 1223.16 last 94.98) - open/close 47093 (time 1787791799.08 total 1233.16 last 84.10) - open/close 48001 (time 1787791809.09 total 1243.17 last 90.70) - open/close 48932 (time 1787791819.10 total 1253.18 last 93.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1260.01 seconds: 39.39 ops/second - unlinked 0 (time 1787791830 ; total 0 ; last 0) - unlinked 10000 (time 1787791948 ; total 118 ; last 118) - unlinked 20000 (time 1787792057 ; total 227 ; last 109) - unlinked 30000 (time 1787792116 ; total 286 ; last 59) - unlinked 40000 (time 1787792180 ; total 350 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 408 seconds: 121.644608 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules via unload_modules_local on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local oleg256-server: modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2