== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:40:30 (1763354430) start mds service on oleg140-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 169 (time 1763354491.21 total 12.21 last 13.84) - open/close 370 (time 1763354501.51 total 22.51 last 19.52) - open/close 484 (time 1763354512.46 total 33.45 last 10.41) - open/close 622 (time 1763354522.50 total 43.50 last 13.74) - open/close 930 (time 1763354533.32 total 54.31 last 28.47) - open/close 1153 (time 1763354543.82 total 64.81 last 21.24) - open/close 1293 (time 1763354553.87 total 74.86 last 13.93) - open/close 1424 (time 1763354564.11 total 85.10 last 12.79) - open/close 1700 (time 1763354574.35 total 95.35 last 26.93) - open/close 1958 (time 1763354584.53 total 105.52 last 25.36) - open/close 2247 (time 1763354594.53 total 115.53 last 28.88) - open/close 2479 (time 1763354604.82 total 125.82 last 22.55) - open/close 2800 (time 1763354615.50 total 136.49 last 30.07) - open/close 2941 (time 1763354625.61 total 146.60 last 13.94) - open/close 3123 (time 1763354635.92 total 156.91 last 17.66) - open/close 3309 (time 1763354646.42 total 167.42 last 17.70) - open/close 3528 (time 1763354657.36 total 178.35 last 20.03) - open/close 3760 (time 1763354667.72 total 188.71 last 22.39) - open/close 3986 (time 1763354678.22 total 199.21 last 21.52) - open/close 4145 (time 1763354688.46 total 209.45 last 15.53) - open/close 4420 (time 1763354699.02 total 220.01 last 26.05) - open/close 4569 (time 1763354709.07 total 230.06 last 14.83) - open/close 4762 (time 1763354719.18 total 240.18 last 19.08) - open/close 4982 (time 1763354729.68 total 250.67 last 20.96) - open/close 5130 (time 1763354739.92 total 260.91 last 14.46) - open/close 5277 (time 1763354749.96 total 270.96 last 14.63) - open/close 5422 (time 1763354760.91 total 281.90 last 13.25) - open/close 5568 (time 1763354771.15 total 292.14 last 14.26) - open/close 5777 (time 1763354781.58 total 302.57 last 20.03) - open/close 5960 (time 1763354791.88 total 312.87 last 17.76) - open/close 6331 (time 1763354802.64 total 323.63 last 34.50) - open/close 6905 (time 1763354812.94 total 333.93 last 55.70) - open/close 7355 (time 1763354823.12 total 344.11 last 44.21) - open/close 7765 (time 1763354833.87 total 354.86 last 38.14) - open/close 8335 (time 1763354843.87 total 364.86 last 56.99) - open/close 10000 (time 1763354850.34 total 371.34 last 257.23) - open/close 13222 (time 1763354860.34 total 381.34 last 322.12) - open/close 16719 (time 1763354870.35 total 391.34 last 349.66) - open/close 19801 (time 1763354880.35 total 401.34 last 308.14) - open/close 22604 (time 1763354890.35 total 411.35 last 280.17) - open/close 25188 (time 1763354900.35 total 421.35 last 258.33) - open/close 27336 (time 1763354910.36 total 431.36 last 214.65) - open/close 29551 (time 1763354920.36 total 441.36 last 221.50) - open/close 32443 (time 1763354930.36 total 451.36 last 289.14) - open/close 35359 (time 1763354940.36 total 461.36 last 291.60) - open/close 38232 (time 1763354950.37 total 471.36 last 287.22) - open/close 40000 (time 1763354957.12 total 478.11 last 261.86) - open/close 42696 (time 1763354967.12 total 488.11 last 269.60) - open/close 44957 (time 1763354977.12 total 498.11 last 226.09) - open/close 47502 (time 1763354987.12 total 508.11 last 254.50) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 516.78 seconds: 96.04 ops/second - unlinked 0 (time 1763354997 ; total 0 ; last 0) - unlinked 10000 (time 1763355019 ; total 22 ; last 22) - unlinked 20000 (time 1763355042 ; total 45 ; last 23) - unlinked 30000 (time 1763355065 ; total 68 ; last 23) - unlinked 40000 (time 1763355084 ; total 87 ; last 19) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 107 seconds: 463.850464 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg140-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2