== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:26:18 (1767320778) start mds service on oleg327-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg327-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 224 (time 1767320941.71 total 10.47 last 21.40) - open/close 517 (time 1767320951.72 total 20.47 last 29.28) - open/close 738 (time 1767320961.74 total 30.49 last 22.06) - open/close 1005 (time 1767320972.24 total 41.00 last 25.42) - open/close 1248 (time 1767320982.30 total 51.06 last 24.15) - open/close 1456 (time 1767320992.97 total 61.73 last 19.50) - open/close 1751 (time 1767321002.98 total 71.74 last 29.47) - open/close 2032 (time 1767321013.64 total 82.39 last 26.37) - open/close 2250 (time 1767321024.59 total 93.34 last 19.91) - open/close 2328 (time 1767321034.59 total 103.34 last 7.80) - open/close 2572 (time 1767321044.87 total 113.63 last 23.72) - open/close 2862 (time 1767321054.88 total 123.63 last 28.99) - open/close 3091 (time 1767321064.89 total 133.65 last 22.87) - open/close 3315 (time 1767321074.89 total 143.65 last 22.40) - open/close 3542 (time 1767321085.46 total 154.21 last 21.49) - open/close 3762 (time 1767321095.72 total 164.48 last 21.43) - open/close 3997 (time 1767321105.73 total 174.49 last 23.48) - open/close 4207 (time 1767321116.50 total 185.26 last 19.50) - open/close 4469 (time 1767321126.81 total 195.57 last 25.41) - open/close 4677 (time 1767321136.98 total 205.73 last 20.46) - open/close 4912 (time 1767321147.84 total 216.59 last 21.63) - open/close 5182 (time 1767321157.92 total 226.67 last 26.79) - open/close 5449 (time 1767321168.48 total 237.24 last 25.27) - open/close 5690 (time 1767321178.96 total 247.71 last 23.01) - open/close 5957 (time 1767321189.80 total 258.56 last 24.61) - open/close 6164 (time 1767321199.82 total 268.58 last 20.66) - open/close 6306 (time 1767321210.25 total 279.00 last 13.62) - open/close 6524 (time 1767321220.63 total 289.38 last 20.99) - open/close 6747 (time 1767321230.93 total 299.68 last 21.65) - open/close 6961 (time 1767321241.24 total 310.00 last 20.75) - open/close 7189 (time 1767321251.79 total 320.55 last 21.62) - open/close 7352 (time 1767321263.38 total 332.13 last 14.07) - open/close 7650 (time 1767321274.36 total 343.11 last 27.13) - open/close 7856 (time 1767321284.39 total 353.14 last 20.53) - open/close 8156 (time 1767321295.07 total 363.82 last 28.09) - open/close 8357 (time 1767321305.68 total 374.44 last 18.94) - open/close 8531 (time 1767321316.36 total 385.11 last 16.30) - open/close 8749 (time 1767321327.27 total 396.03 last 19.98) - open/close 8913 (time 1767321337.36 total 406.12 last 16.25) - open/close 9177 (time 1767321347.38 total 416.13 last 26.37) - open/close 9422 (time 1767321357.70 total 426.45 last 23.73) - open/close 9635 (time 1767321367.83 total 436.58 last 21.03) - open/close 9871 (time 1767321377.85 total 446.61 last 23.54) - open/close 10000 (time 1767321381.91 total 450.67 last 31.77) - open/close 10313 (time 1767321391.93 total 460.69 last 31.23) - open/close 10595 (time 1767321402.58 total 471.33 last 26.50) - open/close 10858 (time 1767321413.38 total 482.13 last 24.34) - open/close 11107 (time 1767321424.77 total 493.53 last 21.86) - open/close 11288 (time 1767321435.49 total 504.24 last 16.89) - open/close 11431 (time 1767321445.77 total 514.53 last 13.90) - open/close 11706 (time 1767321456.53 total 525.29 last 25.56) - open/close 11918 (time 1767321466.53 total 535.29 last 21.20) - open/close 12145 (time 1767321476.74 total 545.50 last 22.23) - open/close 12380 (time 1767321487.01 total 555.77 last 22.89) - open/close 12654 (time 1767321497.04 total 565.79 last 27.33) - open/close 12867 (time 1767321507.22 total 575.97 last 20.93) - open/close 13134 (time 1767321517.70 total 586.45 last 25.47) - open/close 13400 (time 1767321528.31 total 597.06 last 25.08) - open/close 13655 (time 1767321538.31 total 607.07 last 25.48) - open/close 13887 (time 1767321548.51 total 617.27 last 22.74) - open/close 14111 (time 1767321559.37 total 628.13 last 20.63) - open/close 14411 (time 1767321569.60 total 638.36 last 29.32) - open/close 14705 (time 1767321580.17 total 648.93 last 27.82) - open/close 14962 (time 1767321590.19 total 658.94 last 25.67) - open/close 15158 (time 1767321600.52 total 669.28 last 18.96) - open/close 15388 (time 1767321610.52 total 679.28 last 22.99) - open/close 15628 (time 1767321620.53 total 689.29 last 23.98) - open/close 15865 (time 1767321630.54 total 699.29 last 23.69) - open/close 16137 (time 1767321640.55 total 709.31 last 27.17) - open/close 16360 (time 1767321651.43 total 720.18 last 20.51) - open/close 16650 (time 1767321661.71 total 730.46 last 28.21) - open/close 16898 (time 1767321671.81 total 740.57 last 24.54) - open/close 17148 (time 1767321681.82 total 750.57 last 24.99) - open/close 17317 (time 1767321691.82 total 760.58 last 16.89) - open/close 17543 (time 1767321702.73 total 771.49 last 20.71) - open/close 17782 (time 1767321712.80 total 781.55 last 23.74) - open/close 18064 (time 1767321723.40 total 792.16 last 26.60) - open/close 18300 (time 1767321734.28 total 803.03 last 21.70) - open/close 18545 (time 1767321744.28 total 813.04 last 24.49) - open/close 18798 (time 1767321754.66 total 823.41 last 24.38) - open/close 18992 (time 1767321764.84 total 833.59 last 19.05) - open/close 19297 (time 1767321775.73 total 844.49 last 28.00) - open/close 19610 (time 1767321786.77 total 855.52 last 28.37) - open/close 19867 (time 1767321797.76 total 866.52 last 23.37) - open/close 20000 (time 1767321802.80 total 871.56 last 26.38) - open/close 20240 (time 1767321812.83 total 881.59 last 23.93) - open/close 20527 (time 1767321823.31 total 892.06 last 27.40) - open/close 20760 (time 1767321833.35 total 902.11 last 23.19) - open/close 21768 (time 1767321843.36 total 912.11 last 100.74) - open/close 22803 (time 1767321853.36 total 922.12 last 103.49) - open/close 23818 (time 1767321863.36 total 932.12 last 101.49) - open/close 24907 (time 1767321873.36 total 942.12 last 108.89) - open/close 25936 (time 1767321883.37 total 952.12 last 102.87) - open/close 26958 (time 1767321893.37 total 962.13 last 102.14) - open/close 28068 (time 1767321903.37 total 972.13 last 111.00) - open/close 29118 (time 1767321913.38 total 982.13 last 104.96) - open/close 30000 (time 1767321921.27 total 990.03 last 111.67) - open/close 30967 (time 1767321931.29 total 1000.05 last 96.51) - open/close 31848 (time 1767321941.29 total 1010.05 last 88.09) - open/close 32812 (time 1767321951.30 total 1020.05 last 96.36) - open/close 33896 (time 1767321961.30 total 1030.06 last 108.34) - open/close 34874 (time 1767321971.31 total 1040.07 last 97.73) - open/close 35957 (time 1767321981.32 total 1050.07 last 108.23) - open/close 36931 (time 1767321991.33 total 1060.08 last 97.32) - open/close 37797 (time 1767322001.33 total 1070.08 last 86.60) - open/close 38713 (time 1767322011.35 total 1080.10 last 91.39) - open/close 39607 (time 1767322021.35 total 1090.11 last 89.35) - open/close 40000 (time 1767322025.15 total 1093.90 last 103.65) - open/close 41041 (time 1767322035.15 total 1103.90 last 104.09) - open/close 42044 (time 1767322045.15 total 1113.91 last 100.24) - open/close 42964 (time 1767322055.16 total 1123.91 last 91.93) - open/close 43768 (time 1767322065.18 total 1133.93 last 80.28) - open/close 44690 (time 1767322075.18 total 1143.94 last 92.11) - open/close 45595 (time 1767322085.19 total 1153.94 last 90.46) - open/close 46572 (time 1767322095.19 total 1163.94 last 97.69) - open/close 47698 (time 1767322105.19 total 1173.94 last 112.59) - open/close 48620 (time 1767322115.20 total 1183.95 last 92.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1193.69 seconds: 41.58 ops/second - unlinked 0 (time 1767322129 ; total 0 ; last 0) - unlinked 10000 (time 1767322241 ; total 112 ; last 112) - unlinked 20000 (time 1767322337 ; total 208 ; last 96) - unlinked 30000 (time 1767322396 ; total 267 ; last 59) - unlinked 40000 (time 1767322453 ; total 324 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 383 seconds: 129.587463 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg327-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg327-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2