== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:43:18 (1763084598) start mds service on oleg454-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg454-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg454-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1763084656.09 total 10.10 last 16.44) - open/close 354 (time 1763084668.38 total 22.39 last 15.30) - open/close 683 (time 1763084679.01 total 33.01 last 30.96) - open/close 856 (time 1763084689.38 total 43.39 last 16.68) - open/close 1192 (time 1763084700.06 total 54.07 last 31.45) - open/close 1377 (time 1763084710.62 total 64.63 last 17.52) - open/close 1562 (time 1763084721.05 total 75.06 last 17.73) - open/close 1763 (time 1763084731.49 total 85.50 last 19.25) - open/close 1915 (time 1763084741.72 total 95.73 last 14.85) - open/close 2073 (time 1763084751.77 total 105.78 last 15.73) - open/close 2321 (time 1763084764.32 total 118.32 last 19.77) - open/close 2444 (time 1763084774.36 total 128.37 last 12.25) - open/close 2638 (time 1763084784.60 total 138.61 last 18.94) - open/close 2808 (time 1763084794.97 total 148.98 last 16.40) - open/close 2997 (time 1763084805.27 total 159.28 last 18.35) - open/close 3200 (time 1763084815.77 total 169.78 last 19.34) - open/close 3349 (time 1763084826.01 total 180.02 last 14.55) - open/close 3500 (time 1763084836.12 total 190.13 last 14.93) - open/close 3760 (time 1763084846.62 total 200.63 last 24.77) - open/close 3888 (time 1763084857.63 total 211.64 last 11.63) - open/close 4433 (time 1763084867.93 total 221.94 last 52.90) - open/close 4502 (time 1763084878.49 total 232.50 last 6.53) - open/close 4701 (time 1763084888.80 total 242.80 last 19.31) - open/close 4910 (time 1763084899.17 total 253.18 last 20.15) - open/close 5185 (time 1763084909.72 total 263.73 last 26.05) - open/close 5357 (time 1763084919.97 total 273.98 last 16.78) - open/close 5582 (time 1763084930.40 total 284.41 last 21.58) - open/close 5753 (time 1763084941.27 total 295.28 last 15.72) - open/close 6250 (time 1763084952.28 total 306.29 last 45.16) - open/close 6937 (time 1763084962.78 total 316.79 last 65.45) - open/close 7130 (time 1763084972.78 total 326.79 last 19.30) - open/close 7671 (time 1763084982.94 total 336.94 last 53.27) - open/close 8124 (time 1763084993.06 total 347.06 last 44.76) - open/close 8267 (time 1763085003.29 total 357.30 last 13.97) - open/close 8987 (time 1763085013.29 total 367.30 last 71.99) - open/close 10000 (time 1763085016.58 total 370.59 last 307.96) - open/close 12866 (time 1763085026.58 total 380.59 last 286.53) - open/close 15472 (time 1763085036.59 total 390.60 last 260.53) - open/close 17958 (time 1763085046.59 total 400.60 last 248.56) - open/close 20000 (time 1763085052.86 total 406.87 last 325.55) - open/close 22837 (time 1763085062.86 total 416.87 last 283.63) - open/close 25604 (time 1763085072.87 total 426.87 last 276.66) - open/close 28353 (time 1763085082.87 total 436.88 last 274.86) - open/close 30000 (time 1763085088.90 total 442.91 last 273.02) - open/close 32764 (time 1763085098.90 total 452.91 last 276.38) - open/close 35759 (time 1763085108.90 total 462.91 last 299.43) - open/close 38698 (time 1763085118.90 total 472.91 last 293.84) - open/close 40000 (time 1763085123.77 total 477.78 last 267.73) - open/close 41889 (time 1763085133.77 total 487.78 last 188.82) - open/close 43909 (time 1763085143.77 total 497.78 last 201.94) - open/close 46347 (time 1763085153.78 total 507.78 last 243.78) - open/close 48036 (time 1763085163.78 total 517.79 last 168.81) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 525.65 seconds: 94.42 ops/second - unlinked 0 (time 1763085173 ; total 0 ; last 0) - unlinked 10000 (time 1763085199 ; total 26 ; last 26) - unlinked 20000 (time 1763085227 ; total 54 ; last 28) - unlinked 30000 (time 1763085252 ; total 79 ; last 25) - unlinked 40000 (time 1763085275 ; total 102 ; last 23) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 123 seconds: 403.512207 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg454-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg454-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2