== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:56:18 (1763157378) start mds service on oleg443-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg443-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg443-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg443-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 249 (time 1763157441.87 total 10.10 last 24.66) - open/close 528 (time 1763157451.88 total 20.10 last 27.90) - open/close 774 (time 1763157462.42 total 30.64 last 23.33) - open/close 1098 (time 1763157473.11 total 41.33 last 30.32) - open/close 1275 (time 1763157483.41 total 51.63 last 17.18) - open/close 1578 (time 1763157494.35 total 62.58 last 27.69) - open/close 1786 (time 1763157504.36 total 72.58 last 20.79) - open/close 1991 (time 1763157516.50 total 84.72 last 16.88) - open/close 2286 (time 1763157527.39 total 95.61 last 27.10) - open/close 2606 (time 1763157537.39 total 105.61 last 31.99) - open/close 2821 (time 1763157547.41 total 115.63 last 21.46) - open/close 3126 (time 1763157558.29 total 126.51 last 28.05) - open/close 3449 (time 1763157568.91 total 137.13 last 30.40) - open/close 3706 (time 1763157579.41 total 147.63 last 24.48) - open/close 4003 (time 1763157589.97 total 158.20 last 28.11) - open/close 4231 (time 1763157599.98 total 168.20 last 22.79) - open/close 4527 (time 1763157610.90 total 179.13 last 27.09) - open/close 4816 (time 1763157621.84 total 190.06 last 26.43) - open/close 5048 (time 1763157631.89 total 200.12 last 23.07) - open/close 5367 (time 1763157642.84 total 211.07 last 29.13) - open/close 5663 (time 1763157653.59 total 221.81 last 27.55) - open/close 5981 (time 1763157663.59 total 231.82 last 31.79) - open/close 6200 (time 1763157674.20 total 242.42 last 20.65) - open/close 6468 (time 1763157684.50 total 252.72 last 26.02) - open/close 6639 (time 1763157695.26 total 263.48 last 15.90) - open/close 6812 (time 1763157705.44 total 273.66 last 16.99) - open/close 7050 (time 1763157715.86 total 284.08 last 22.84) - open/close 7380 (time 1763157725.91 total 294.13 last 32.83) - open/close 7434 (time 1763157736.85 total 305.07 last 4.94) - open/close 7615 (time 1763157747.16 total 315.38 last 17.56) - open/close 7736 (time 1763157757.71 total 325.94 last 11.46) - open/close 7974 (time 1763157768.02 total 336.24 last 23.10) - open/close 8228 (time 1763157778.33 total 346.55 last 24.63) - open/close 8538 (time 1763157789.01 total 357.23 last 29.02) - open/close 8830 (time 1763157799.70 total 367.92 last 27.31) - open/close 9128 (time 1763157810.64 total 378.86 last 27.24) - open/close 9441 (time 1763157821.59 total 389.81 last 28.59) - open/close 9709 (time 1763157832.54 total 400.77 last 24.46) - open/close 9898 (time 1763157842.64 total 410.86 last 18.72) - open/close 10000 (time 1763157846.62 total 414.84 last 25.66) - open/close 10198 (time 1763157856.66 total 424.88 last 19.71) - open/close 10416 (time 1763157867.41 total 435.63 last 20.28) - open/close 10680 (time 1763157878.04 total 446.26 last 24.84) - open/close 11004 (time 1763157889.04 total 457.26 last 29.45) - open/close 11525 (time 1763157899.05 total 467.27 last 52.06) - open/close 12484 (time 1763157909.05 total 477.27 last 95.88) - open/close 13570 (time 1763157919.05 total 487.27 last 108.59) - open/close 14789 (time 1763157929.06 total 497.28 last 121.79) - open/close 15999 (time 1763157939.07 total 507.29 last 120.90) - open/close 17252 (time 1763157949.07 total 517.29 last 125.29) - open/close 18498 (time 1763157959.07 total 527.29 last 124.59) - open/close 20000 (time 1763157967.52 total 535.74 last 177.87) - open/close 21851 (time 1763157977.52 total 545.74 last 185.04) - open/close 23775 (time 1763157987.52 total 555.75 last 192.34) - open/close 25626 (time 1763157997.52 total 565.75 last 185.05) - open/close 27380 (time 1763158007.53 total 575.75 last 175.37) - open/close 29286 (time 1763158017.53 total 585.75 last 190.51) - open/close 30000 (time 1763158022.37 total 590.59 last 147.60) - open/close 31633 (time 1763158032.37 total 600.59 last 163.29) - open/close 33128 (time 1763158042.37 total 610.59 last 149.48) - open/close 35151 (time 1763158052.38 total 620.60 last 202.21) - open/close 37023 (time 1763158062.38 total 630.60 last 187.12) - open/close 38742 (time 1763158072.38 total 640.61 last 171.84) - open/close 40000 (time 1763158078.69 total 646.91 last 199.52) - open/close 42053 (time 1763158088.69 total 656.91 last 205.29) - open/close 44018 (time 1763158098.69 total 666.92 last 196.40) - open/close 45752 (time 1763158108.69 total 676.92 last 173.40) - open/close 47989 (time 1763158118.69 total 686.92 last 223.67) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 694.83 seconds: 71.43 ops/second - unlinked 0 (time 1763158128 ; total 0 ; last 0) - unlinked 10000 (time 1763158171 ; total 43 ; last 43) - unlinked 20000 (time 1763158204 ; total 76 ; last 33) - unlinked 30000 (time 1763158234 ; total 106 ; last 30) - unlinked 40000 (time 1763158266 ; total 138 ; last 32) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 165 seconds: 300.799988 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local oleg443-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg443-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2