== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:53:08 (1764744788) Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1811 (time 1764744841.80 total 10.00 last 181.02) - open/close 3773 (time 1764744851.80 total 20.01 last 196.17) - open/close 6014 (time 1764744861.81 total 30.01 last 223.96) - open/close 7762 (time 1764744871.81 total 40.01 last 174.75) - open/close 10000 (time 1764744881.60 total 49.80 last 228.64) - open/close 12520 (time 1764744891.60 total 59.81 last 251.91) - open/close 14938 (time 1764744901.60 total 69.81 last 241.70) - open/close 18191 (time 1764744911.61 total 79.81 last 325.27) - open/close 20000 (time 1764744918.25 total 86.46 last 272.30) - open/close 22861 (time 1764744928.25 total 96.46 last 286.06) - open/close 26169 (time 1764744938.25 total 106.46 last 330.77) - open/close 29944 (time 1764744948.25 total 116.46 last 377.47) - open/close 33773 (time 1764744958.25 total 126.46 last 382.88) - open/close 38000 (time 1764744968.25 total 136.46 last 422.68) - open/close 40000 (time 1764744973.43 total 141.63 last 386.55) - open/close 44390 (time 1764744983.43 total 151.63 last 438.95) - open/close 48513 (time 1764744993.43 total 161.64 last 412.23) - open/close 50000 (time 1764744996.74 total 164.94 last 449.96) - open/close 54182 (time 1764745006.74 total 174.94 last 418.20) - open/close 58262 (time 1764745016.74 total 184.94 last 407.97) - open/close 60000 (time 1764745020.69 total 188.90 last 439.28) - open/close 63686 (time 1764745030.69 total 198.90 last 368.53) total: 64768 open/close in 202.48 seconds: 319.87 ops/second - /unlink 7778 (time 1764745045.90 total 10.00 last 777.75) - /unlink 10000 (time 1764745048.77 total 12.87 last 774.20) - /unlink 17043 (time 1764745058.77 total 22.87 last 704.24) - /unlink 20000 (time 1764745063.06 total 27.16 last 689.74) - /unlink 26984 (time 1764745073.06 total 37.16 last 698.37) - /unlink 30000 (time 1764745076.94 total 41.03 last 778.31) - /unlink 37345 (time 1764745086.94 total 51.04 last 734.42) - /unlink 40000 (time 1764745090.38 total 54.48 last 770.81) - /unlink 47463 (time 1764745100.38 total 64.48 last 746.23) - /unlink 50000 (time 1764745103.66 total 67.75 last 774.93) - /unlink 58240 (time 1764745113.66 total 77.75 last 823.96) - /unlink 60000 (time 1764745115.96 total 80.05 last 765.20) total: 64768 /unlink in 85.68 seconds: 755.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4065 (time 1764745157.04 total 10.00 last 406.43) - open/close 8894 (time 1764745167.04 total 20.00 last 482.84) - open/close 10000 (time 1764745169.21 total 22.17 last 509.70) - open/close 13949 (time 1764745179.21 total 32.17 last 394.84) - open/close 18206 (time 1764745189.21 total 42.18 last 425.65) - open/close 20000 (time 1764745193.13 total 46.09 last 458.27) - open/close 23951 (time 1764745203.13 total 56.09 last 395.06) - open/close 28892 (time 1764745213.13 total 66.09 last 494.04) - open/close 30000 (time 1764745215.18 total 68.14 last 540.56) - open/close 34690 (time 1764745225.18 total 78.14 last 468.95) - open/close 39505 (time 1764745235.18 total 88.14 last 481.49) - open/close 43876 (time 1764745245.18 total 98.14 last 437.10) - open/close 48315 (time 1764745255.20 total 108.16 last 443.08) - open/close 50000 (time 1764745258.84 total 111.81 last 462.37) - open/close 54491 (time 1764745268.84 total 121.81 last 449.08) - open/close 59347 (time 1764745278.84 total 131.81 last 485.53) - open/close 63731 (time 1764745288.85 total 141.81 last 438.34) total: 64768 open/close in 144.12 seconds: 449.41 ops/second - /unlink 7427 (time 1764745302.51 total 10.00 last 742.68) - /unlink 10000 (time 1764745305.75 total 13.24 last 793.82) - /unlink 17988 (time 1764745315.75 total 23.24 last 798.77) - /unlink 20000 (time 1764745318.87 total 26.36 last 644.74) - /unlink 27900 (time 1764745328.88 total 36.36 last 789.93) - /unlink 30000 (time 1764745331.06 total 38.55 last 962.32) - /unlink 38793 (time 1764745341.06 total 48.55 last 879.26) - /unlink 47393 (time 1764745351.06 total 58.55 last 860.00) - /unlink 50000 (time 1764745354.20 total 61.69 last 830.14) - /unlink 58354 (time 1764745364.20 total 71.69 last 835.35) total: 64768 /unlink in 79.21 seconds: 817.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4666 (time 1764745471.56 total 10.00 last 466.59) - open/close 9331 (time 1764745481.56 total 20.00 last 466.45) - open/close 13866 (time 1764745491.56 total 30.00 last 453.40) - open/close 18194 (time 1764745501.56 total 40.01 last 432.70) - open/close 20000 (time 1764745505.01 total 43.45 last 524.67) - open/close 24587 (time 1764745515.01 total 53.45 last 458.67) - open/close 29109 (time 1764745525.01 total 63.45 last 452.05) - open/close 30000 (time 1764745527.06 total 65.50 last 435.25) - open/close 34500 (time 1764745537.06 total 75.50 last 449.91) - open/close 39060 (time 1764745547.06 total 85.50 last 455.87) - open/close 40000 (time 1764745549.33 total 87.77 last 414.01) - open/close 44867 (time 1764745559.33 total 97.78 last 486.66) - open/close 49872 (time 1764745569.33 total 107.78 last 500.43) - open/close 54326 (time 1764745579.34 total 117.78 last 445.10) - open/close 58793 (time 1764745589.34 total 127.78 last 446.70) - open/close 60000 (time 1764745592.01 total 130.45 last 451.93) - open/close 64448 (time 1764745602.01 total 140.45 last 444.80) total: 64768 open/close in 141.24 seconds: 458.58 ops/second - /unlink 7362 (time 1764745614.21 total 10.00 last 736.20) - /unlink 10000 (time 1764745618.11 total 13.90 last 675.80) - /unlink 17069 (time 1764745628.11 total 23.90 last 706.83) - /unlink 20000 (time 1764745631.96 total 27.75 last 761.35) - /unlink 27189 (time 1764745641.96 total 37.76 last 718.84) - /unlink 30000 (time 1764745645.74 total 41.53 last 743.77) - /unlink 37408 (time 1764745655.74 total 51.54 last 740.76) - /unlink 40000 (time 1764745659.37 total 55.16 last 714.61) - /unlink 47507 (time 1764745669.37 total 65.16 last 750.66) - /unlink 50000 (time 1764745672.47 total 68.26 last 804.62) - /unlink 57529 (time 1764745682.47 total 78.26 last 752.88) - /unlink 60000 (time 1764745686.02 total 81.81 last 696.84) total: 64768 /unlink in 88.13 seconds: 734.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4761 (time 1764745725.41 total 10.00 last 476.01) - open/close 9380 (time 1764745735.41 total 20.00 last 461.88) - open/close 13664 (time 1764745745.41 total 30.00 last 428.39) - open/close 17919 (time 1764745755.41 total 40.00 last 425.50) - open/close 20000 (time 1764745760.26 total 44.85 last 429.02) - open/close 24072 (time 1764745770.26 total 54.85 last 407.17) - open/close 28211 (time 1764745780.26 total 64.85 last 413.87) - open/close 30000 (time 1764745784.56 total 69.15 last 416.64) - open/close 34365 (time 1764745794.56 total 79.15 last 436.45) - open/close 38856 (time 1764745804.58 total 89.17 last 448.03) - open/close 40000 (time 1764745807.31 total 91.90 last 419.29) - open/close 44062 (time 1764745817.31 total 101.90 last 406.18) - open/close 48263 (time 1764745827.31 total 111.90 last 420.02) - open/close 50000 (time 1764745831.59 total 116.18 last 406.22) - open/close 54152 (time 1764745841.59 total 126.18 last 415.15) - open/close 58143 (time 1764745851.59 total 136.18 last 399.09) - open/close 60000 (time 1764745856.03 total 140.62 last 418.55) - open/close 64366 (time 1764745866.03 total 150.62 last 436.54) total: 64768 open/close in 151.56 seconds: 427.33 ops/second - /unlink 7968 (time 1764745878.31 total 10.00 last 796.78) - /unlink 10000 (time 1764745881.19 total 12.88 last 704.42) - /unlink 13318 (time 1764745891.19 total 22.89 last 331.76) - /unlink 15445 (time 1764745901.20 total 32.89 last 212.60) - /unlink 20000 (time 1764745909.78 total 41.48 last 530.54) - /unlink 28161 (time 1764745919.78 total 51.48 last 816.10) - /unlink 30000 (time 1764745922.38 total 54.07 last 708.97) - /unlink 38408 (time 1764745932.38 total 64.07 last 840.63) - /unlink 45586 (time 1764745942.38 total 74.07 last 717.78) - /unlink 50000 (time 1764745948.34 total 80.03 last 740.78) - /unlink 57942 (time 1764745958.34 total 90.03 last 794.14) - /unlink 60000 (time 1764745960.53 total 92.23 last 937.22) total: 64768 /unlink in 97.98 seconds: 661.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4752 (time 1764745994.16 total 10.00 last 475.09) - open/close 9943 (time 1764746004.16 total 20.00 last 519.09) - open/close 14519 (time 1764746014.17 total 30.00 last 457.58) - open/close 19173 (time 1764746024.17 total 40.00 last 465.32) - open/close 24398 (time 1764746034.17 total 50.01 last 522.41) - open/close 29820 (time 1764746044.17 total 60.01 last 542.17) - open/close 34936 (time 1764746054.17 total 70.01 last 511.60) - open/close 39438 (time 1764746064.18 total 80.01 last 449.87) - open/close 43703 (time 1764746074.18 total 90.02 last 426.45) - open/close 48016 (time 1764746084.18 total 100.02 last 431.25) - open/close 50000 (time 1764746088.94 total 104.78 last 416.55) - open/close 54345 (time 1764746098.94 total 114.78 last 434.41) - open/close 58613 (time 1764746108.94 total 124.78 last 426.78) - open/close 60000 (time 1764746112.42 total 128.26 last 398.58) - open/close 64602 (time 1764746122.43 total 138.26 last 460.13) total: 64768 open/close in 138.58 seconds: 467.38 ops/second - /unlink 8443 (time 1764746134.08 total 10.00 last 844.28) - /unlink 16973 (time 1764746144.08 total 20.00 last 852.97) - /unlink 20000 (time 1764746148.21 total 24.13 last 733.13) - /unlink 27902 (time 1764746158.21 total 34.13 last 790.17) - /unlink 30000 (time 1764746160.94 total 36.86 last 768.49) - /unlink 37778 (time 1764746170.94 total 46.86 last 777.72) - /unlink 40000 (time 1764746173.59 total 49.52 last 836.50) - /unlink 47949 (time 1764746183.59 total 59.52 last 794.89) - /unlink 50000 (time 1764746186.21 total 62.14 last 782.87) - /unlink 57845 (time 1764746196.21 total 72.14 last 784.39) - /unlink 60000 (time 1764746199.00 total 74.93 last 772.96) total: 64768 /unlink in 80.71 seconds: 802.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2