== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:14:44 (1763352884) start mds service on oleg213-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg213-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 253 (time 1763352926.46 total 10.17 last 24.87) - open/close 498 (time 1763352936.57 total 20.29 last 24.22) - open/close 752 (time 1763352947.00 total 30.72 last 24.35) - open/close 962 (time 1763352957.49 total 41.21 last 20.01) - open/close 1153 (time 1763352968.18 total 51.90 last 17.87) - open/close 1338 (time 1763352978.43 total 62.14 last 18.06) - open/close 1614 (time 1763352988.67 total 72.38 last 26.95) - open/close 1883 (time 1763352999.61 total 83.32 last 24.59) - open/close 2141 (time 1763353010.61 total 94.33 last 23.43) - open/close 2370 (time 1763353021.37 total 105.08 last 21.30) - open/close 2582 (time 1763353032.12 total 115.83 last 19.73) - open/close 2747 (time 1763353042.49 total 126.20 last 15.91) - open/close 2938 (time 1763353053.05 total 136.76 last 18.09) - open/close 3072 (time 1763353063.16 total 146.88 last 13.25) - open/close 3336 (time 1763353073.98 total 157.69 last 24.41) - open/close 3527 (time 1763353084.53 total 168.25 last 18.09) - open/close 3639 (time 1763353095.35 total 179.06 last 10.36) - open/close 3821 (time 1763353105.66 total 189.37 last 17.66) - open/close 4070 (time 1763353116.40 total 200.12 last 23.16) - open/close 4270 (time 1763353127.09 total 210.81 last 18.71) - open/close 4431 (time 1763353137.27 total 220.99 last 15.81) - open/close 4640 (time 1763353148.02 total 231.74 last 19.44) - open/close 4818 (time 1763353158.26 total 241.98 last 17.38) - open/close 5052 (time 1763353169.14 total 252.86 last 21.51) - open/close 5253 (time 1763353179.57 total 263.29 last 19.27) - open/close 5483 (time 1763353190.45 total 274.17 last 21.14) - open/close 5693 (time 1763353201.08 total 284.79 last 19.77) - open/close 5960 (time 1763353211.12 total 294.84 last 26.58) - open/close 6155 (time 1763353221.69 total 305.40 last 18.46) - open/close 6340 (time 1763353231.93 total 315.64 last 18.07) - open/close 6509 (time 1763353241.98 total 325.69 last 16.82) - open/close 6785 (time 1763353251.98 total 335.69 last 27.60) - open/close 7040 (time 1763353262.90 total 346.62 last 23.34) - open/close 7183 (time 1763353272.95 total 356.67 last 14.23) - open/close 7472 (time 1763353283.20 total 366.92 last 28.20) - open/close 7764 (time 1763353294.20 total 377.91 last 26.55) - open/close 8034 (time 1763353305.07 total 388.79 last 24.82) - open/close 8957 (time 1763353315.08 total 398.79 last 92.27) - open/close 10000 (time 1763353319.58 total 403.30 last 231.50) - open/close 11570 (time 1763353329.59 total 413.30 last 156.95) - open/close 13851 (time 1763353339.59 total 423.31 last 228.05) - open/close 16314 (time 1763353349.59 total 433.31 last 246.27) - open/close 18240 (time 1763353359.59 total 443.31 last 192.57) - open/close 20000 (time 1763353368.13 total 451.85 last 206.16) - open/close 22318 (time 1763353378.13 total 461.85 last 231.72) - open/close 24580 (time 1763353388.13 total 471.85 last 226.14) - open/close 26880 (time 1763353398.14 total 481.85 last 229.93) - open/close 28724 (time 1763353408.14 total 491.86 last 184.32) - open/close 30000 (time 1763353414.65 total 498.36 last 196.20) - open/close 31700 (time 1763353424.65 total 508.36 last 169.96) - open/close 33414 (time 1763353434.65 total 518.37 last 171.37) - open/close 35700 (time 1763353444.65 total 528.37 last 228.57) - open/close 37603 (time 1763353454.65 total 538.37 last 190.24) - open/close 39993 (time 1763353464.67 total 548.39 last 238.52) - open/close 42283 (time 1763353474.68 total 558.39 last 228.94) - open/close 44152 (time 1763353484.68 total 568.40 last 186.81) - open/close 46322 (time 1763353494.68 total 578.40 last 216.94) - open/close 48201 (time 1763353504.68 total 588.40 last 187.90) total: 50000 open/close in 595.88 seconds: 83.91 ops/second - unlinked 0 (time 1763353513 ; total 0 ; last 0) - unlinked 10000 (time 1763353534 ; total 21 ; last 21) - unlinked 20000 (time 1763353558 ; total 45 ; last 24) - unlinked 30000 (time 1763353580 ; total 67 ; last 22) - unlinked 40000 (time 1763353602 ; total 89 ; last 22) total: 50000 unlinks in 108 seconds: 462.962952 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg213-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2