== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:41:06 (1764794466) start mds service on oleg224-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 598 (time 1764794523.20 total 10.10 last 59.20) - open/close 1162 (time 1764794533.89 total 20.79 last 52.76) - open/close 1772 (time 1764794544.00 total 30.90 last 60.35) - open/close 2358 (time 1764794554.30 total 41.21 last 56.86) - open/close 2830 (time 1764794565.19 total 52.09 last 43.38) - open/close 3515 (time 1764794575.68 total 62.58 last 65.25) - open/close 4095 (time 1764794587.78 total 74.68 last 47.96) - open/close 4684 (time 1764794597.82 total 84.72 last 58.63) - open/close 5306 (time 1764794607.94 total 94.84 last 61.49) - open/close 5643 (time 1764794617.94 total 104.84 last 33.69) - open/close 5897 (time 1764794627.94 total 114.84 last 25.40) - open/close 6095 (time 1764794637.98 total 124.88 last 19.73) - open/close 6303 (time 1764794647.98 total 134.88 last 20.80) - open/close 6542 (time 1764794657.99 total 144.89 last 23.89) - open/close 6705 (time 1764794668.44 total 155.34 last 15.60) - open/close 6920 (time 1764794678.54 total 165.44 last 21.28) - open/close 7044 (time 1764794688.55 total 175.45 last 12.39) - open/close 7290 (time 1764794698.63 total 185.53 last 24.39) - open/close 7578 (time 1764794709.64 total 196.54 last 26.17) - open/close 7829 (time 1764794720.46 total 207.36 last 23.19) - open/close 7984 (time 1764794730.47 total 217.37 last 15.49) - open/close 8106 (time 1764794740.47 total 227.37 last 12.20) - open/close 8275 (time 1764794750.47 total 237.37 last 16.89) - open/close 8593 (time 1764794760.48 total 247.38 last 31.78) - open/close 8801 (time 1764794770.49 total 257.39 last 20.78) - open/close 8884 (time 1764794780.88 total 267.78 last 7.98) - open/close 9017 (time 1764794791.24 total 278.14 last 12.84) - open/close 9063 (time 1764794801.84 total 288.74 last 4.34) - open/close 9132 (time 1764794812.45 total 299.35 last 6.50) - open/close 9259 (time 1764794822.62 total 309.52 last 12.49) - open/close 9491 (time 1764794833.25 total 320.15 last 21.83) - open/close 9658 (time 1764794843.25 total 330.15 last 16.69) - open/close 9915 (time 1764794853.98 total 340.88 last 23.97) - open/close 10000 (time 1764794857.62 total 344.52 last 23.31) - open/close 10222 (time 1764794867.63 total 354.53 last 22.19) - open/close 10433 (time 1764794878.54 total 365.45 last 19.32) - open/close 10618 (time 1764794888.55 total 375.45 last 18.49) - open/close 10775 (time 1764794899.46 total 386.36 last 14.39) - open/close 10951 (time 1764794909.58 total 396.48 last 17.40) - open/close 11189 (time 1764794920.35 total 407.25 last 22.10) - open/close 11427 (time 1764794930.78 total 417.68 last 22.82) - open/close 11631 (time 1764794941.28 total 428.18 last 19.42) - open/close 11745 (time 1764794951.29 total 438.19 last 11.40) - open/close 11971 (time 1764794962.20 total 449.10 last 20.71) - open/close 12273 (time 1764794972.74 total 459.64 last 28.65) - open/close 12498 (time 1764794983.37 total 470.27 last 21.17) - open/close 12726 (time 1764794993.42 total 480.32 last 22.69) - open/close 12929 (time 1764795005.78 total 492.68 last 16.42) - open/close 13089 (time 1764795016.21 total 503.11 last 15.35) - open/close 13213 (time 1764795026.95 total 513.85 last 11.54) - open/close 13396 (time 1764795036.97 total 523.87 last 18.27) - open/close 13495 (time 1764795047.83 total 534.73 last 9.12) - open/close 13646 (time 1764795057.83 total 544.73 last 15.09) - open/close 13816 (time 1764795067.84 total 554.74 last 16.99) - open/close 13838 (time 1764795077.87 total 564.77 last 2.19) - open/close 13982 (time 1764795087.95 total 574.85 last 14.29) - open/close 14168 (time 1764795097.96 total 584.86 last 18.57) - open/close 14523 (time 1764795107.97 total 594.87 last 35.48) - open/close 15253 (time 1764795117.97 total 604.87 last 72.99) - open/close 15970 (time 1764795127.98 total 614.88 last 71.65) - open/close 16667 (time 1764795137.98 total 624.88 last 69.67) - open/close 17406 (time 1764795147.99 total 634.89 last 73.85) - open/close 18174 (time 1764795157.99 total 644.89 last 76.79) - open/close 18951 (time 1764795168.00 total 654.90 last 77.64) - open/close 19722 (time 1764795178.00 total 664.90 last 77.09) - open/close 20000 (time 1764795181.69 total 668.59 last 75.35) - open/close 20748 (time 1764795191.69 total 678.59 last 74.77) - open/close 21465 (time 1764795201.70 total 688.61 last 71.61) - open/close 22164 (time 1764795211.71 total 698.61 last 69.83) - open/close 22880 (time 1764795221.72 total 708.62 last 71.56) - open/close 23579 (time 1764795231.72 total 718.62 last 69.90) - open/close 24331 (time 1764795241.72 total 728.62 last 75.19) - open/close 25082 (time 1764795251.74 total 738.64 last 74.98) - open/close 25874 (time 1764795261.75 total 748.65 last 79.11) - open/close 26587 (time 1764795271.75 total 758.66 last 71.26) - open/close 27328 (time 1764795281.77 total 768.67 last 73.99) - open/close 28072 (time 1764795291.78 total 778.68 last 74.36) - open/close 28820 (time 1764795301.78 total 788.68 last 74.74) - open/close 29523 (time 1764795311.80 total 798.70 last 70.21) - open/close 30000 (time 1764795318.45 total 805.35 last 71.73) - open/close 30735 (time 1764795328.45 total 815.35 last 73.44) - open/close 31418 (time 1764795338.46 total 825.36 last 68.25) - open/close 32097 (time 1764795348.46 total 835.36 last 67.89) - open/close 32722 (time 1764795358.47 total 845.37 last 62.44) - open/close 33335 (time 1764795368.48 total 855.38 last 61.24) - open/close 33958 (time 1764795378.49 total 865.39 last 62.25) - open/close 34504 (time 1764795388.49 total 875.39 last 54.58) - open/close 35070 (time 1764795398.50 total 885.41 last 56.54) - open/close 35790 (time 1764795408.51 total 895.41 last 71.96) - open/close 36472 (time 1764795418.53 total 905.43 last 68.09) - open/close 37116 (time 1764795428.54 total 915.44 last 64.34) - open/close 37794 (time 1764795438.54 total 925.44 last 67.79) - open/close 38410 (time 1764795448.54 total 935.44 last 61.57) - open/close 38973 (time 1764795458.55 total 945.45 last 56.24) - open/close 39587 (time 1764795468.56 total 955.46 last 61.38) - open/close 40000 (time 1764795475.66 total 962.56 last 58.14) - open/close 40666 (time 1764795485.67 total 972.57 last 66.56) - open/close 41282 (time 1764795495.67 total 982.57 last 61.59) - open/close 41892 (time 1764795505.68 total 992.58 last 60.92) - open/close 42570 (time 1764795515.69 total 1002.59 last 67.75) - open/close 43194 (time 1764795525.70 total 1012.60 last 62.35) - open/close 43776 (time 1764795535.71 total 1022.61 last 58.14) - open/close 44417 (time 1764795545.71 total 1032.61 last 64.08) - open/close 45053 (time 1764795555.72 total 1042.62 last 63.55) - open/close 45722 (time 1764795565.72 total 1052.62 last 66.88) - open/close 46351 (time 1764795575.74 total 1062.64 last 62.79) - open/close 47036 (time 1764795585.76 total 1072.66 last 68.34) - open/close 47739 (time 1764795595.79 total 1082.69 last 70.13) - open/close 48432 (time 1764795605.79 total 1092.69 last 69.29) - open/close 49090 (time 1764795615.81 total 1102.72 last 65.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1110.66 seconds: 44.69 ops/second - unlinked 0 (time 1764795628 ; total 0 ; last 0) - unlinked 10000 (time 1764795774 ; total 146 ; last 146) - unlinked 20000 (time 1764795911 ; total 283 ; last 137) - unlinked 30000 (time 1764796056 ; total 428 ; last 145) - unlinked 40000 (time 1764796200 ; total 572 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 709 seconds: 70.002823 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2