== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:32:04 (1755639124) start mds service on oleg355-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg355-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions oleg355-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg355-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg355-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 320 (time 1755639165.90 total 10.06 last 31.81) - open/close 591 (time 1755639176.85 total 21.00 last 24.76) - open/close 802 (time 1755639187.73 total 31.88 last 19.39) - open/close 1265 (time 1755639198.10 total 42.25 last 44.66) - open/close 1775 (time 1755639208.53 total 52.68 last 48.88) - open/close 2304 (time 1755639219.09 total 63.24 last 50.10) - open/close 2789 (time 1755639229.52 total 73.68 last 46.48) - open/close 3095 (time 1755639239.52 total 83.68 last 30.59) - open/close 3322 (time 1755639250.51 total 94.67 last 20.66) - open/close 3539 (time 1755639262.48 total 106.64 last 18.13) - open/close 4041 (time 1755639272.98 total 117.13 last 47.83) - open/close 4415 (time 1755639283.34 total 127.50 last 36.07) - open/close 4841 (time 1755639293.97 total 138.12 last 40.10) - open/close 5110 (time 1755639304.02 total 148.17 last 26.77) - open/close 5577 (time 1755639314.58 total 158.73 last 44.21) - open/close 6026 (time 1755639325.14 total 169.29 last 42.53) - open/close 6386 (time 1755639335.38 total 179.53 last 35.15) - open/close 6766 (time 1755639345.55 total 189.71 last 37.35) - open/close 7068 (time 1755639355.55 total 199.71 last 30.20) - open/close 7382 (time 1755639365.55 total 209.71 last 31.39) - open/close 7807 (time 1755639375.89 total 220.04 last 41.13) - open/close 8272 (time 1755639386.13 total 230.29 last 45.40) - open/close 8594 (time 1755639396.13 total 240.29 last 32.19) - open/close 8954 (time 1755639406.29 total 250.45 last 35.44) - open/close 9241 (time 1755639416.29 total 260.45 last 28.70) - open/close 9604 (time 1755639426.38 total 270.54 last 35.97) - open/close 10000 (time 1755639435.59 total 279.75 last 42.99) - open/close 12574 (time 1755639445.60 total 289.75 last 257.37) - open/close 16917 (time 1755639455.60 total 299.75 last 434.26) - open/close 20000 (time 1755639462.66 total 306.82 last 436.40) - open/close 24550 (time 1755639472.66 total 316.82 last 454.93) - open/close 28603 (time 1755639482.66 total 326.82 last 405.27) - open/close 30000 (time 1755639486.03 total 330.19 last 414.67) - open/close 34449 (time 1755639496.03 total 340.19 last 444.83) - open/close 38945 (time 1755639506.04 total 350.19 last 449.55) - open/close 40000 (time 1755639508.56 total 352.72 last 417.54) - open/close 44261 (time 1755639518.56 total 362.72 last 426.02) - open/close 48707 (time 1755639528.56 total 372.72 last 444.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 374.87 seconds: 132.40 ops/second - unlinked 0 (time 1755639531 ; total 0 ; last 0) - unlinked 10000 (time 1755639544 ; total 13 ; last 13) - unlinked 20000 (time 1755639557 ; total 26 ; last 13) - unlinked 30000 (time 1755639570 ; total 39 ; last 13) - unlinked 40000 (time 1755639583 ; total 52 ; last 13) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 64 seconds: 775.500000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg355-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2