== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:22:32 (1769757752) start mds service on oleg357-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 283 (time 1769757845.34 total 10.00 last 28.29) - open/close 544 (time 1769757856.22 total 20.88 last 24.00) - open/close 809 (time 1769757866.22 total 30.88 last 26.50) - open/close 1042 (time 1769757876.69 total 41.36 last 22.24) - open/close 1299 (time 1769757887.27 total 51.93 last 24.30) - open/close 1503 (time 1769757898.27 total 62.93 last 18.55) - open/close 1753 (time 1769757909.11 total 73.77 last 23.07) - open/close 2099 (time 1769757919.11 total 83.77 last 34.59) - open/close 2311 (time 1769757929.83 total 94.49 last 19.78) - open/close 2552 (time 1769757939.93 total 104.59 last 23.86) - open/close 2813 (time 1769757949.93 total 114.59 last 26.09) - open/close 3047 (time 1769757960.49 total 125.15 last 22.17) - open/close 3301 (time 1769757971.03 total 135.69 last 24.09) - open/close 3477 (time 1769757981.59 total 146.25 last 16.67) - open/close 3721 (time 1769757991.90 total 156.56 last 23.66) - open/close 3839 (time 1769758002.78 total 167.45 last 10.84) - open/close 4016 (time 1769758013.21 total 177.87 last 16.97) - open/close 4235 (time 1769758024.10 total 188.76 last 20.12) - open/close 4385 (time 1769758034.26 total 198.93 last 14.75) - open/close 4592 (time 1769758044.46 total 209.12 last 20.30) - open/close 4845 (time 1769758055.46 total 220.12 last 23.01) - open/close 5030 (time 1769758066.46 total 231.13 last 16.80) - open/close 5250 (time 1769758076.46 total 241.13 last 22.00) - open/close 5454 (time 1769758086.47 total 251.13 last 20.39) - open/close 5659 (time 1769758097.32 total 261.98 last 18.89) - open/close 5894 (time 1769758108.25 total 272.91 last 21.50) - open/close 6050 (time 1769758119.27 total 283.93 last 14.16) - open/close 6197 (time 1769758129.46 total 294.12 last 14.43) - open/close 6413 (time 1769758140.06 total 304.72 last 20.37) - open/close 6600 (time 1769758150.30 total 314.96 last 18.27) - open/close 6808 (time 1769758160.67 total 325.33 last 20.06) - open/close 6932 (time 1769758173.72 total 338.39 last 9.50) - open/close 7063 (time 1769758183.79 total 348.45 last 13.02) - open/close 7223 (time 1769758194.27 total 358.93 last 15.27) - open/close 7396 (time 1769758204.83 total 369.49 last 16.37) - open/close 7545 (time 1769758214.84 total 379.50 last 14.89) - open/close 7699 (time 1769758225.24 total 389.90 last 14.81) - open/close 7854 (time 1769758235.42 total 400.08 last 15.22) - open/close 8103 (time 1769758246.42 total 411.08 last 22.63) - open/close 8243 (time 1769758256.43 total 421.09 last 13.98) - open/close 8381 (time 1769758267.16 total 431.82 last 12.87) - open/close 8527 (time 1769758278.11 total 442.78 last 13.33) - open/close 8650 (time 1769758288.86 total 453.53 last 11.44) - open/close 8801 (time 1769758299.75 total 464.41 last 13.87) - open/close 8957 (time 1769758310.68 total 475.34 last 14.28) - open/close 9220 (time 1769758321.17 total 485.83 last 25.05) - open/close 9422 (time 1769758331.23 total 495.89 last 20.09) - open/close 9606 (time 1769758341.27 total 505.93 last 18.32) - open/close 9803 (time 1769758352.03 total 516.69 last 18.31) - open/close 9983 (time 1769758362.33 total 527.00 last 17.47) - open/close 10095 (time 1769758373.14 total 537.80 last 10.36) - open/close 10238 (time 1769758383.39 total 548.05 last 13.96) - open/close 10465 (time 1769758393.40 total 558.06 last 22.68) - open/close 10612 (time 1769758404.32 total 568.98 last 13.46) - open/close 10700 (time 1769758414.56 total 579.22 last 8.59) - open/close 10925 (time 1769758425.57 total 590.23 last 20.43) - open/close 11093 (time 1769758435.61 total 600.27 last 16.73) - open/close 11232 (time 1769758445.66 total 610.32 last 13.83) - open/close 11391 (time 1769758456.49 total 621.15 last 14.68) - open/close 11501 (time 1769758466.92 total 631.58 last 10.54) - open/close 11957 (time 1769758476.93 total 641.59 last 45.58) - open/close 13194 (time 1769758486.93 total 651.59 last 123.63) - open/close 14422 (time 1769758496.94 total 661.60 last 122.73) - open/close 15708 (time 1769758506.94 total 671.60 last 128.56) - open/close 17089 (time 1769758516.94 total 681.60 last 138.10) - open/close 18224 (time 1769758526.94 total 691.60 last 113.49) - open/close 19463 (time 1769758536.95 total 701.61 last 123.81) - open/close 20000 (time 1769758541.37 total 706.03 last 121.58) - open/close 21212 (time 1769758551.37 total 716.03 last 121.13) - open/close 22554 (time 1769758561.37 total 726.04 last 134.16) - open/close 23879 (time 1769758571.38 total 736.04 last 132.41) - open/close 25115 (time 1769758581.38 total 746.05 last 123.55) - open/close 26309 (time 1769758591.39 total 756.06 last 119.29) - open/close 27562 (time 1769758601.40 total 766.06 last 125.28) - open/close 28897 (time 1769758611.40 total 776.06 last 133.42) - open/close 30000 (time 1769758620.14 total 784.80 last 126.22) - open/close 31407 (time 1769758630.15 total 794.81 last 140.62) - open/close 32727 (time 1769758640.15 total 804.81 last 131.97) - open/close 34060 (time 1769758650.15 total 814.81 last 133.27) - open/close 35754 (time 1769758660.15 total 824.81 last 169.38) - open/close 37312 (time 1769758670.15 total 834.82 last 155.77) - open/close 38954 (time 1769758680.16 total 844.82 last 164.14) - open/close 40000 (time 1769758687.18 total 851.85 last 148.86) - open/close 41518 (time 1769758697.20 total 861.86 last 151.63) - open/close 42894 (time 1769758707.20 total 871.86 last 137.59) - open/close 44466 (time 1769758717.20 total 881.86 last 157.13) - open/close 46034 (time 1769758727.21 total 891.87 last 156.60) - open/close 47318 (time 1769758737.22 total 901.88 last 128.34) - open/close 48742 (time 1769758747.22 total 911.88 last 142.37) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 918.37 seconds: 54.04 ops/second - unlinked 0 (time 1769758755 ; total 0 ; last 0) - unlinked 10000 (time 1769758821 ; total 66 ; last 66) - unlinked 20000 (time 1769758860 ; total 105 ; last 39) - unlinked 30000 (time 1769758898 ; total 143 ; last 38) - unlinked 40000 (time 1769758932 ; total 177 ; last 34) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 210 seconds: 236.342850 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg357-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2