== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:50:33 (1773237033) start mds service on oleg428-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 696 (time 1773237075.60 total 10.65 last 65.38) - open/close 1356 (time 1773237086.35 total 21.40 last 61.38) - open/close 2115 (time 1773237097.30 total 32.34 last 69.35) - open/close 2742 (time 1773237107.92 total 42.97 last 59.02) - open/close 3305 (time 1773237118.23 total 53.27 last 54.62) - open/close 3982 (time 1773237128.91 total 63.96 last 63.36) - open/close 4682 (time 1773237139.86 total 74.91 last 63.92) - open/close 5432 (time 1773237150.87 total 85.91 last 68.16) - open/close 6020 (time 1773237161.29 total 96.34 last 56.39) - open/close 6536 (time 1773237171.60 total 106.65 last 50.07) - open/close 7057 (time 1773237181.97 total 117.01 last 50.26) - open/close 7622 (time 1773237192.53 total 127.58 last 53.50) - open/close 8413 (time 1773237202.53 total 137.58 last 79.08) - open/close 9216 (time 1773237212.53 total 147.58 last 80.30) - open/close 9902 (time 1773237223.57 total 158.61 last 62.16) - open/close 10604 (time 1773237233.57 total 168.61 last 70.20) - open/close 11233 (time 1773237244.24 total 179.29 last 58.92) - open/close 12037 (time 1773237255.25 total 190.30 last 73.06) - open/close 12792 (time 1773237266.19 total 201.24 last 68.99) - open/close 13394 (time 1773237276.62 total 211.67 last 57.70) - open/close 13914 (time 1773237287.05 total 222.10 last 49.86) - open/close 14641 (time 1773237297.74 total 232.79 last 68.00) - open/close 15353 (time 1773237308.43 total 243.48 last 66.63) - open/close 16004 (time 1773237318.99 total 254.04 last 61.64) - open/close 16650 (time 1773237329.55 total 264.60 last 61.17) - open/close 17273 (time 1773237340.05 total 275.09 last 59.37) - open/close 17792 (time 1773237350.42 total 285.47 last 50.04) - open/close 18260 (time 1773237360.59 total 295.64 last 46.00) - open/close 20000 (time 1773237365.14 total 300.19 last 382.12) - open/close 25272 (time 1773237375.14 total 310.19 last 527.16) - open/close 30000 (time 1773237384.06 total 319.11 last 530.44) - open/close 35247 (time 1773237394.07 total 329.11 last 524.25) - open/close 40000 (time 1773237403.42 total 338.47 last 507.97) - open/close 45022 (time 1773237413.42 total 348.47 last 502.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49730) error: No space left on device total: 49730 open/close in 357.19 seconds: 139.23 ops/second - unlinked 0 (time 1773237423 ; total 0 ; last 0) - unlinked 10000 (time 1773237435 ; total 12 ; last 12) - unlinked 20000 (time 1773237446 ; total 23 ; last 11) - unlinked 30000 (time 1773237458 ; total 35 ; last 12) - unlinked 40000 (time 1773237468 ; total 45 ; last 10) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49730) error: No such file or directory total: 49730 unlinks in 55 seconds: 904.181824 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg428-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2