== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:47:57 (1769082477) start mds service on oleg431-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 637 (time 1769082519.54 total 10.75 last 59.28) - open/close 1223 (time 1769082530.17 total 21.37 last 55.16) - open/close 1726 (time 1769082540.41 total 31.61 last 49.12) - open/close 2367 (time 1769082551.10 total 42.30 last 59.98) - open/close 2895 (time 1769082561.59 total 52.80 last 50.29) - open/close 3449 (time 1769082572.22 total 63.42 last 52.16) - open/close 3890 (time 1769082582.39 total 73.59 last 43.34) - open/close 4312 (time 1769082592.63 total 83.84 last 41.20) - open/close 4685 (time 1769082603.64 total 94.84 last 33.88) - open/close 5106 (time 1769082613.64 total 104.85 last 42.09) - open/close 5863 (time 1769082624.51 total 115.71 last 69.69) - open/close 6533 (time 1769082635.26 total 126.46 last 62.33) - open/close 7131 (time 1769082645.94 total 137.15 last 55.95) - open/close 7948 (time 1769082656.89 total 148.09 last 74.64) - open/close 8657 (time 1769082667.77 total 158.97 last 65.18) - open/close 9289 (time 1769082678.52 total 169.72 last 58.77) - open/close 10000 (time 1769082688.31 total 179.52 last 72.61) - open/close 10668 (time 1769082699.07 total 190.27 last 62.13) - open/close 11368 (time 1769082709.94 total 201.15 last 64.35) - open/close 11803 (time 1769082720.12 total 211.32 last 42.74) - open/close 12289 (time 1769082730.42 total 221.63 last 47.17) - open/close 12593 (time 1769082741.43 total 232.64 last 27.62) - open/close 12949 (time 1769082751.54 total 242.75 last 35.21) - open/close 13321 (time 1769082761.66 total 252.86 last 36.78) - open/close 13875 (time 1769082772.09 total 263.29 last 53.11) - open/close 14284 (time 1769082782.20 total 273.40 last 40.44) - open/close 14637 (time 1769082793.21 total 284.42 last 32.05) - open/close 16991 (time 1769082803.22 total 294.42 last 235.37) - open/close 20000 (time 1769082809.64 total 300.84 last 468.70) - open/close 24552 (time 1769082819.64 total 310.84 last 455.16) - open/close 29208 (time 1769082829.64 total 320.84 last 465.59) - open/close 34161 (time 1769082839.64 total 330.84 last 495.22) - open/close 39116 (time 1769082849.64 total 340.84 last 495.39) - open/close 43809 (time 1769082859.64 total 350.84 last 469.22) - open/close 48530 (time 1769082869.64 total 360.85 last 472.02) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 363.13 seconds: 136.68 ops/second - unlinked 0 (time 1769082873 ; total 0 ; last 0) - unlinked 10000 (time 1769082885 ; total 12 ; last 12) - unlinked 20000 (time 1769082897 ; total 24 ; last 12) - unlinked 30000 (time 1769082909 ; total 36 ; last 12) - unlinked 40000 (time 1769082920 ; total 47 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 58 seconds: 855.724121 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg431-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2