== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:08:05 (1784941685) start mds service on oleg151-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg151-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1784941755.65 total 10.10 last 23.57) - open/close 441 (time 1784941766.27 total 20.72 last 19.11) - open/close 640 (time 1784941776.77 total 31.22 last 18.95) - open/close 846 (time 1784941787.28 total 41.73 last 19.61) - open/close 1066 (time 1784941797.98 total 52.43 last 20.55) - open/close 1281 (time 1784941808.65 total 63.10 last 20.15) - open/close 1460 (time 1784941818.66 total 73.11 last 17.89) - open/close 1662 (time 1784941829.32 total 83.77 last 18.95) - open/close 1856 (time 1784941839.46 total 93.91 last 19.14) - open/close 2039 (time 1784941849.46 total 103.91 last 18.29) - open/close 2250 (time 1784941860.10 total 114.55 last 19.83) - open/close 2433 (time 1784941870.61 total 125.06 last 17.41) - open/close 2630 (time 1784941880.77 total 135.22 last 19.39) - open/close 2856 (time 1784941891.60 total 146.05 last 20.87) - open/close 3025 (time 1784941901.81 total 156.25 last 16.56) - open/close 3255 (time 1784941912.39 total 166.84 last 21.73) - open/close 3524 (time 1784941922.76 total 177.21 last 25.94) - open/close 3698 (time 1784941932.77 total 187.22 last 17.37) - open/close 3810 (time 1784941942.79 total 197.24 last 11.19) - open/close 3988 (time 1784941952.80 total 207.25 last 17.78) - open/close 4155 (time 1784941963.72 total 218.17 last 15.29) - open/close 4337 (time 1784941974.66 total 229.11 last 16.64) - open/close 4525 (time 1784941985.23 total 239.68 last 17.78) - open/close 4697 (time 1784941995.97 total 250.42 last 16.02) - open/close 4880 (time 1784942005.97 total 260.42 last 18.30) - open/close 5068 (time 1784942016.03 total 270.48 last 18.69) - open/close 5300 (time 1784942026.96 total 281.41 last 21.23) - open/close 5468 (time 1784942037.59 total 292.03 last 15.81) - open/close 5615 (time 1784942047.83 total 302.28 last 14.35) - open/close 5788 (time 1784942057.83 total 312.28 last 17.29) - open/close 5966 (time 1784942067.86 total 322.31 last 17.76) - open/close 6113 (time 1784942077.91 total 332.35 last 14.63) - open/close 6305 (time 1784942088.07 total 342.52 last 18.89) - open/close 6469 (time 1784942098.69 total 353.14 last 15.44) - open/close 6595 (time 1784942108.69 total 363.14 last 12.60) - open/close 6778 (time 1784942119.37 total 373.82 last 17.13) - open/close 6917 (time 1784942129.38 total 383.83 last 13.89) - open/close 7053 (time 1784942140.23 total 394.68 last 12.53) - open/close 7140 (time 1784942150.79 total 405.24 last 8.24) - open/close 7284 (time 1784942161.04 total 415.49 last 14.04) - open/close 7391 (time 1784942172.03 total 426.48 last 9.74) - open/close 7629 (time 1784942182.08 total 436.52 last 23.69) - open/close 7759 (time 1784942192.21 total 446.66 last 12.83) - open/close 7910 (time 1784942202.56 total 457.01 last 14.58) - open/close 8026 (time 1784942213.51 total 467.96 last 10.60) - open/close 8136 (time 1784942224.26 total 478.70 last 10.24) - open/close 8224 (time 1784942234.75 total 489.20 last 8.39) - open/close 8331 (time 1784942244.75 total 499.20 last 10.70) - open/close 8526 (time 1784942254.75 total 509.20 last 19.49) - open/close 8728 (time 1784942264.83 total 519.28 last 20.04) - open/close 8869 (time 1784942275.15 total 529.59 last 13.67) - open/close 8997 (time 1784942285.31 total 539.76 last 12.59) - open/close 9153 (time 1784942295.45 total 549.89 last 15.39) - open/close 9271 (time 1784942306.31 total 560.76 last 10.86) - open/close 9552 (time 1784942316.62 total 571.07 last 27.27) - open/close 9771 (time 1784942327.44 total 581.89 last 20.24) - open/close 9917 (time 1784942337.92 total 592.37 last 13.93) - open/close 10000 (time 1784942343.19 total 597.64 last 15.74) - open/close 10208 (time 1784942353.20 total 607.65 last 20.78) - open/close 10398 (time 1784942364.23 total 618.68 last 17.22) - open/close 10636 (time 1784942374.86 total 629.31 last 22.40) - open/close 10779 (time 1784942385.28 total 639.73 last 13.72) - open/close 10953 (time 1784942396.05 total 650.50 last 16.16) - open/close 11084 (time 1784942406.28 total 660.73 last 12.81) - open/close 11986 (time 1784942416.29 total 670.74 last 90.10) - open/close 12945 (time 1784942426.29 total 680.74 last 95.86) - open/close 14021 (time 1784942436.30 total 690.75 last 107.55) - open/close 15210 (time 1784942446.30 total 700.75 last 118.85) - open/close 16272 (time 1784942456.31 total 710.75 last 106.16) - open/close 17186 (time 1784942466.31 total 720.76 last 91.34) - open/close 18308 (time 1784942476.32 total 730.76 last 112.17) - open/close 19299 (time 1784942486.33 total 740.78 last 98.97) - open/close 20000 (time 1784942494.12 total 748.57 last 89.97) - open/close 21023 (time 1784942504.12 total 758.57 last 102.28) - open/close 22049 (time 1784942514.12 total 768.57 last 102.60) - open/close 23073 (time 1784942524.13 total 778.58 last 102.35) - open/close 24118 (time 1784942534.14 total 788.59 last 104.36) - open/close 25076 (time 1784942544.14 total 798.59 last 95.80) - open/close 26069 (time 1784942554.15 total 808.59 last 99.25) - open/close 27064 (time 1784942564.16 total 818.60 last 99.39) - open/close 28246 (time 1784942574.16 total 828.61 last 118.13) - open/close 29329 (time 1784942584.17 total 838.62 last 108.18) - open/close 30000 (time 1784942590.23 total 844.67 last 110.88) - open/close 31099 (time 1784942600.23 total 854.68 last 109.80) - open/close 32215 (time 1784942610.24 total 864.69 last 111.49) - open/close 33375 (time 1784942620.25 total 874.70 last 115.94) - open/close 34971 (time 1784942630.25 total 884.70 last 159.54) - open/close 36199 (time 1784942640.25 total 894.70 last 122.80) - open/close 37359 (time 1784942650.26 total 904.71 last 115.90) - open/close 38458 (time 1784942660.31 total 914.75 last 109.41) - open/close 39620 (time 1784942670.31 total 924.76 last 116.19) - open/close 40000 (time 1784942673.93 total 928.38 last 104.96) - open/close 40995 (time 1784942683.94 total 938.38 last 99.42) - open/close 42073 (time 1784942693.94 total 948.38 last 107.80) - open/close 43115 (time 1784942703.94 total 958.39 last 104.15) - open/close 44186 (time 1784942713.95 total 968.40 last 107.02) - open/close 45207 (time 1784942723.95 total 978.40 last 102.03) - open/close 46193 (time 1784942733.96 total 988.40 last 98.58) - open/close 47237 (time 1784942743.96 total 998.41 last 104.36) - open/close 48162 (time 1784942753.96 total 1008.41 last 92.46) - open/close 49196 (time 1784942763.97 total 1018.42 last 103.35) total: 50000 open/close in 1026.69 seconds: 48.70 ops/second - unlinked 0 (time 1784942774 ; total 0 ; last 0) - unlinked 10000 (time 1784942853 ; total 79 ; last 79) - unlinked 20000 (time 1784942952 ; total 178 ; last 99) - unlinked 30000 (time 1784943032 ; total 258 ; last 80) - unlinked 40000 (time 1784943110 ; total 336 ; last 78) total: 50000 unlinks in 414 seconds: 120.772949 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local oleg151-server: modules unloaded. pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2