== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:21:19 (1767036079) start mds service on oleg330-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions oleg330-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg330-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 652 (time 1767036121.05 total 10.06 last 64.80) - open/close 1329 (time 1767036131.05 total 20.06 last 67.69) - open/close 1963 (time 1767036141.15 total 30.16 last 62.81) - open/close 2679 (time 1767036151.78 total 40.78 last 67.38) - open/close 3188 (time 1767036162.72 total 51.73 last 46.51) - open/close 3792 (time 1767036172.96 total 61.97 last 59.00) - open/close 4414 (time 1767036183.84 total 72.85 last 57.15) - open/close 5077 (time 1767036193.84 total 82.85 last 66.29) - open/close 5643 (time 1767036204.51 total 93.52 last 53.06) - open/close 6317 (time 1767036215.39 total 104.40 last 61.93) - open/close 6961 (time 1767036226.27 total 115.28 last 59.20) - open/close 7625 (time 1767036237.21 total 126.22 last 60.68) - open/close 8250 (time 1767036248.09 total 137.10 last 57.44) - open/close 8902 (time 1767036258.20 total 147.21 last 64.49) - open/close 9622 (time 1767036268.57 total 157.58 last 69.45) - open/close 10000 (time 1767036273.69 total 162.70 last 73.80) - open/close 10612 (time 1767036283.70 total 172.71 last 61.19) - open/close 11262 (time 1767036293.70 total 182.71 last 64.99) - open/close 11932 (time 1767036303.90 total 192.91 last 65.66) - open/close 12642 (time 1767036314.91 total 203.92 last 64.49) - open/close 13331 (time 1767036324.91 total 213.92 last 68.90) - open/close 14036 (time 1767036335.26 total 224.27 last 68.10) - open/close 14817 (time 1767036345.63 total 234.64 last 75.31) - open/close 15578 (time 1767036356.06 total 245.07 last 72.95) - open/close 16304 (time 1767036366.07 total 255.08 last 72.57) - open/close 16963 (time 1767036377.12 total 266.13 last 59.62) - open/close 17602 (time 1767036388.00 total 277.00 last 58.76) - open/close 18254 (time 1767036398.00 total 287.01 last 65.20) - open/close 18849 (time 1767036408.00 total 297.01 last 59.48) - open/close 19262 (time 1767036418.65 total 307.66 last 38.77) - open/close 20000 (time 1767036425.02 total 314.03 last 115.82) - open/close 23166 (time 1767036435.02 total 324.03 last 316.59) - open/close 25941 (time 1767036445.03 total 334.04 last 277.47) - open/close 28949 (time 1767036455.03 total 344.04 last 300.72) - open/close 30000 (time 1767036459.00 total 348.01 last 264.75) - open/close 32608 (time 1767036469.00 total 358.01 last 260.71) - open/close 35660 (time 1767036479.00 total 368.01 last 305.12) - open/close 38988 (time 1767036489.01 total 378.02 last 332.74) - open/close 40000 (time 1767036492.11 total 381.12 last 325.59) - open/close 43772 (time 1767036502.11 total 391.12 last 377.18) - open/close 47097 (time 1767036512.12 total 401.12 last 332.48) total: 50000 open/close in 409.63 seconds: 122.06 ops/second - unlinked 0 (time 1767036522 ; total 0 ; last 0) - unlinked 10000 (time 1767036538 ; total 16 ; last 16) - unlinked 20000 (time 1767036556 ; total 34 ; last 18) - unlinked 30000 (time 1767036574 ; total 52 ; last 18) - unlinked 40000 (time 1767036590 ; total 68 ; last 16) total: 50000 unlinks in 82 seconds: 609.756104 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local oleg330-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg330-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2