== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:29:57 (1760138997) start mds service on oleg218-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 445 (time 1760139039.55 total 10.39 last 42.82) - open/close 828 (time 1760139049.60 total 20.44 last 38.11) - open/close 1220 (time 1760139059.60 total 30.44 last 39.19) - open/close 1747 (time 1760139069.88 total 40.73 last 51.23) - open/close 2132 (time 1760139080.57 total 51.42 last 36.03) - open/close 2588 (time 1760139090.57 total 61.42 last 45.59) - open/close 3045 (time 1760139100.86 total 71.70 last 44.42) - open/close 3498 (time 1760139110.97 total 81.82 last 44.79) - open/close 3842 (time 1760139121.47 total 92.31 last 32.78) - open/close 4245 (time 1760139132.22 total 103.06 last 37.48) - open/close 4644 (time 1760139142.22 total 113.07 last 39.89) - open/close 5112 (time 1760139153.02 total 123.87 last 43.34) - open/close 5560 (time 1760139163.02 total 133.87 last 44.79) - open/close 6047 (time 1760139173.24 total 144.09 last 47.65) - open/close 6629 (time 1760139183.48 total 154.33 last 56.83) - open/close 7239 (time 1760139193.98 total 164.82 last 58.12) - open/close 7819 (time 1760139204.73 total 175.58 last 53.94) - open/close 8222 (time 1760139215.42 total 186.27 last 37.69) - open/close 8706 (time 1760139229.18 total 200.02 last 35.19) - open/close 9159 (time 1760139239.18 total 210.02 last 45.30) - open/close 9717 (time 1760139249.85 total 220.69 last 52.30) - open/close 10000 (time 1760139252.65 total 223.50 last 100.82) - open/close 10488 (time 1760139262.66 total 233.50 last 48.79) - open/close 11056 (time 1760139272.66 total 243.51 last 56.79) - open/close 11614 (time 1760139282.81 total 253.66 last 54.98) - open/close 12126 (time 1760139292.81 total 263.66 last 51.19) - open/close 12572 (time 1760139302.82 total 273.66 last 44.58) - open/close 13150 (time 1760139312.82 total 283.66 last 57.79) - open/close 13372 (time 1760139323.97 total 294.81 last 19.91) - open/close 13933 (time 1760139334.01 total 304.86 last 55.85) - open/close 14152 (time 1760139344.01 total 314.86 last 21.90) - open/close 14746 (time 1760139354.23 total 325.08 last 58.10) - open/close 15335 (time 1760139365.24 total 336.09 last 53.51) - open/close 15650 (time 1760139375.93 total 346.78 last 29.47) - open/close 16134 (time 1760139385.98 total 356.83 last 48.17) - open/close 16558 (time 1760139396.22 total 367.07 last 41.39) - open/close 17134 (time 1760139406.78 total 377.62 last 54.57) - open/close 17576 (time 1760139416.89 total 387.74 last 43.71) - open/close 18546 (time 1760139426.90 total 397.74 last 96.96) - open/close 20000 (time 1760139432.78 total 403.63 last 247.05) - open/close 22407 (time 1760139442.78 total 413.63 last 240.65) - open/close 24741 (time 1760139452.78 total 423.63 last 233.40) - open/close 27551 (time 1760139462.79 total 433.63 last 280.94) - open/close 29850 (time 1760139472.79 total 443.63 last 229.87) - open/close 32165 (time 1760139482.79 total 453.63 last 231.48) - open/close 34548 (time 1760139492.79 total 463.63 last 238.26) - open/close 36820 (time 1760139502.79 total 473.63 last 227.20) - open/close 39165 (time 1760139512.79 total 483.64 last 234.42) - open/close 40000 (time 1760139516.17 total 487.02 last 247.17) - open/close 42760 (time 1760139526.17 total 497.02 last 275.98) - open/close 45490 (time 1760139536.17 total 507.02 last 272.95) - open/close 48238 (time 1760139546.20 total 517.04 last 274.21) total: 50000 open/close in 524.00 seconds: 95.42 ops/second - unlinked 0 (time 1760139555 ; total 0 ; last 0) - unlinked 10000 (time 1760139573 ; total 18 ; last 18) - unlinked 20000 (time 1760139593 ; total 38 ; last 20) - unlinked 30000 (time 1760139617 ; total 62 ; last 24) - unlinked 40000 (time 1760139636 ; total 81 ; last 19) total: 50000 unlinks in 100 seconds: 500.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2