== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:08:26 (1756678106) start mds service on oleg144-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg144-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg144-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 637 (time 1756678153.55 total 10.65 last 59.79) - open/close 1274 (time 1756678164.17 total 21.28 last 59.97) - open/close 2049 (time 1756678175.12 total 32.22 last 70.81) - open/close 2751 (time 1756678186.00 total 43.10 last 64.53) - open/close 3432 (time 1756678196.88 total 53.98 last 62.57) - open/close 4159 (time 1756678207.76 total 64.86 last 66.84) - open/close 4891 (time 1756678218.64 total 75.74 last 67.28) - open/close 5631 (time 1756678229.45 total 86.56 last 68.41) - open/close 6343 (time 1756678240.33 total 97.44 last 65.45) - open/close 7091 (time 1756678251.28 total 108.38 last 68.33) - open/close 7701 (time 1756678261.90 total 119.00 last 57.43) - open/close 8249 (time 1756678272.41 total 129.51 last 52.17) - open/close 8941 (time 1756678283.28 total 140.38 last 63.65) - open/close 9426 (time 1756678293.65 total 150.75 last 46.76) - open/close 9822 (time 1756678303.89 total 160.99 last 38.68) - open/close 10000 (time 1756678306.34 total 163.44 last 72.67) - open/close 10627 (time 1756678317.01 total 174.11 last 58.76) - open/close 11300 (time 1756678327.82 total 184.92 last 62.22) - open/close 11860 (time 1756678338.32 total 195.42 last 53.34) - open/close 12379 (time 1756678348.75 total 205.85 last 49.75) - open/close 13095 (time 1756678359.76 total 216.86 last 65.03) - open/close 13696 (time 1756678370.38 total 227.48 last 56.59) - open/close 14383 (time 1756678381.13 total 238.24 last 63.90) - open/close 15039 (time 1756678391.89 total 248.99 last 61.00) - open/close 15753 (time 1756678402.70 total 259.80 last 66.03) - open/close 16274 (time 1756678413.20 total 270.30 last 49.62) - open/close 16787 (time 1756678423.70 total 280.80 last 48.88) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-16913) error: No space left on device total: 16913 open/close in 282.13 seconds: 59.95 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8861: kill: (111494) - No such process - unlinked 0 (time 1756678427 ; total 0 ; last 0) - unlinked 10000 (time 1756678439 ; total 12 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-16913) error: No such file or directory total: 16913 unlinks in 21 seconds: 805.380981 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg144-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg144-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2