== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:44:14 (1760129054) start mds service on oleg412-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg412-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg412-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 138 (time 1760129122.58 total 10.19 last 13.54) - open/close 338 (time 1760129133.15 total 20.76 last 18.92) - open/close 447 (time 1760129143.88 total 31.50 last 10.16) - open/close 619 (time 1760129153.88 total 41.50 last 17.20) - open/close 858 (time 1760129164.55 total 52.17 last 22.40) - open/close 1128 (time 1760129175.06 total 62.68 last 25.69) - open/close 1407 (time 1760129185.94 total 73.55 last 25.65) - open/close 1616 (time 1760129197.57 total 85.19 last 17.96) - open/close 1855 (time 1760129207.58 total 95.19 last 23.89) - open/close 1940 (time 1760129218.06 total 105.67 last 8.11) - open/close 2104 (time 1760129228.24 total 115.85 last 16.11) - open/close 2281 (time 1760129238.28 total 125.90 last 17.62) - open/close 2436 (time 1760129248.65 total 136.27 last 14.94) - open/close 2568 (time 1760129258.70 total 146.32 last 13.13) - open/close 2678 (time 1760129269.58 total 157.20 last 10.11) - open/close 2788 (time 1760129280.47 total 168.09 last 10.10) - open/close 2914 (time 1760129290.48 total 178.10 last 12.59) - open/close 3098 (time 1760129300.49 total 188.11 last 18.38) - open/close 3247 (time 1760129311.00 total 198.61 last 14.19) - open/close 3403 (time 1760129321.23 total 208.85 last 15.24) - open/close 3532 (time 1760129332.24 total 219.85 last 11.72) - open/close 3663 (time 1760129342.61 total 230.22 last 12.63) - open/close 3812 (time 1760129353.16 total 240.78 last 14.11) - open/close 3943 (time 1760129363.28 total 250.89 last 12.95) - open/close 4062 (time 1760129373.51 total 261.13 last 11.62) - open/close 4345 (time 1760129383.52 total 271.13 last 28.28) - open/close 4490 (time 1760129393.53 total 281.14 last 14.49) - open/close 4630 (time 1760129403.54 total 291.16 last 13.98) - open/close 4729 (time 1760129414.10 total 301.71 last 9.38) - open/close 4894 (time 1760129424.72 total 312.33 last 15.53) - open/close 5040 (time 1760129437.72 total 325.33 last 11.23) - open/close 5106 (time 1760129447.77 total 335.39 last 6.56) - open/close 5332 (time 1760129458.65 total 346.26 last 20.78) - open/close 5598 (time 1760129469.40 total 357.02 last 24.74) - open/close 5873 (time 1760129481.62 total 369.23 last 22.51) - open/close 6081 (time 1760129491.97 total 379.59 last 20.08) - open/close 6336 (time 1760129502.67 total 390.28 last 23.84) - open/close 6485 (time 1760129513.51 total 401.12 last 13.74) - open/close 6650 (time 1760129524.62 total 412.24 last 14.85) - open/close 6870 (time 1760129535.13 total 422.75 last 20.93) - open/close 7157 (time 1760129545.14 total 432.76 last 28.68) - open/close 7377 (time 1760129555.47 total 443.08 last 21.30) - open/close 7667 (time 1760129566.22 total 453.83 last 26.98) - open/close 7835 (time 1760129577.30 total 464.91 last 15.16) - open/close 8093 (time 1760129588.19 total 475.80 last 23.70) - open/close 8418 (time 1760129599.17 total 486.79 last 29.58) - open/close 8762 (time 1760129609.67 total 497.29 last 32.76) - open/close 9044 (time 1760129619.72 total 507.34 last 28.07) - open/close 9307 (time 1760129630.41 total 518.03 last 24.61) - open/close 9492 (time 1760129640.41 total 528.03 last 18.49) - open/close 9967 (time 1760129650.42 total 538.03 last 47.48) - open/close 11177 (time 1760129660.42 total 548.03 last 120.99) - open/close 12429 (time 1760129670.42 total 558.04 last 125.19) - open/close 13748 (time 1760129680.42 total 568.04 last 131.88) - open/close 15038 (time 1760129690.43 total 578.04 last 128.94) - open/close 16421 (time 1760129700.43 total 588.05 last 138.20) - open/close 17536 (time 1760129710.44 total 598.06 last 111.41) - open/close 18676 (time 1760129720.44 total 608.06 last 113.99) - open/close 20000 (time 1760129730.42 total 618.04 last 132.69) - open/close 21355 (time 1760129740.42 total 628.04 last 135.46) - open/close 22597 (time 1760129750.43 total 638.04 last 124.14) - open/close 23679 (time 1760129760.43 total 648.05 last 108.13) - open/close 25144 (time 1760129770.44 total 658.05 last 146.43) - open/close 26189 (time 1760129780.45 total 668.07 last 104.36) - open/close 27159 (time 1760129790.45 total 678.07 last 96.99) - open/close 28371 (time 1760129800.45 total 688.07 last 121.18) - open/close 29793 (time 1760129810.46 total 698.08 last 142.09) - open/close 31143 (time 1760129820.47 total 708.08 last 134.91) - open/close 32771 (time 1760129830.47 total 718.09 last 162.76) - open/close 34408 (time 1760129840.47 total 728.09 last 163.68) - open/close 36230 (time 1760129850.48 total 738.09 last 182.14) - open/close 37660 (time 1760129860.48 total 748.10 last 142.91) - open/close 39245 (time 1760129870.49 total 758.10 last 158.46) - open/close 40000 (time 1760129876.54 total 764.15 last 124.78) - open/close 41709 (time 1760129886.54 total 774.15 last 170.85) - open/close 43146 (time 1760129896.54 total 784.15 last 143.68) - open/close 44556 (time 1760129906.54 total 794.16 last 140.99) - open/close 46291 (time 1760129916.55 total 804.16 last 173.40) - open/close 48011 (time 1760129926.55 total 814.17 last 171.92) - open/close 49208 (time 1760129936.56 total 824.17 last 119.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 827.08 seconds: 60.01 ops/second - unlinked 0 (time 1760129941 ; total 0 ; last 0) - unlinked 10000 (time 1760129979 ; total 38 ; last 38) - unlinked 20000 (time 1760130006 ; total 65 ; last 27) - unlinked 30000 (time 1760130039 ; total 98 ; last 33) - unlinked 40000 (time 1760130075 ; total 134 ; last 36) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 169 seconds: 293.680481 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg412-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2