== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:39:26 (1787751566) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg352-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg352-server oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg352-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 636 (time 1787751786.34 total 10.01 last 63.56) - open/close 1194 (time 1787751796.36 total 20.03 last 55.70) - open/close 1839 (time 1787751806.37 total 30.04 last 64.41) - open/close 2478 (time 1787751816.39 total 40.06 last 63.79) - open/close 3154 (time 1787751826.39 total 50.06 last 67.60) - open/close 3774 (time 1787751836.41 total 60.08 last 61.83) - open/close 4437 (time 1787751846.42 total 70.09 last 66.29) total: 4500 open/close in 70.93 seconds: 63.44 ops/second - /unlink 1262 (time 1787751861.57 total 10.00 last 126.20) - /unlink 2416 (time 1787751871.58 total 20.01 last 115.34) - /unlink 3566 (time 1787751881.58 total 30.01 last 114.91) total: 4500 /unlink in 37.96 seconds: 118.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 610 (time 1787751905.26 total 10.00 last 60.98) - open/close 1149 (time 1787751915.27 total 20.01 last 53.86) - open/close 1657 (time 1787751925.28 total 30.02 last 50.77) - open/close 2281 (time 1787751935.29 total 40.03 last 62.33) - open/close 3048 (time 1787751945.29 total 50.04 last 76.65) - open/close 3742 (time 1787751955.30 total 60.04 last 69.38) - open/close 4425 (time 1787751965.30 total 70.05 last 68.25) total: 4500 open/close in 71.37 seconds: 63.05 ops/second - /unlink 1067 (time 1787751982.99 total 10.01 last 106.63) - /unlink 2374 (time 1787751992.99 total 20.01 last 130.65) - /unlink 3645 (time 1787752003.00 total 30.01 last 127.07) total: 4500 /unlink in 36.37 seconds: 123.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 685 (time 1787752025.01 total 10.01 last 68.46) - open/close 1343 (time 1787752035.03 total 20.02 last 65.72) - open/close 2062 (time 1787752045.04 total 30.03 last 71.80) - open/close 2760 (time 1787752055.06 total 40.05 last 69.64) - open/close 3333 (time 1787752065.07 total 50.06 last 57.28) - open/close 3958 (time 1787752075.08 total 60.07 last 62.41) total: 4500 open/close in 68.30 seconds: 65.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1259 (time 1787752098.35 total 10.00 last 125.90) - /unlink 2416 (time 1787752108.37 total 20.01 last 115.58) - /unlink 3711 (time 1787752118.37 total 30.02 last 129.39) total: 4500 /unlink in 36.41 seconds: 123.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 545 (time 1787752140.69 total 10.01 last 54.45) - open/close 1117 (time 1787752150.69 total 20.01 last 57.16) - open/close 1773 (time 1787752160.70 total 30.02 last 65.56) - open/close 2474 (time 1787752170.71 total 40.03 last 70.05) - open/close 3254 (time 1787752180.72 total 50.04 last 77.93) - open/close 3998 (time 1787752190.73 total 60.06 last 74.27) total: 4500 open/close in 67.57 seconds: 66.59 ops/second - /unlink 1340 (time 1787752212.79 total 10.00 last 133.99) - /unlink 2621 (time 1787752222.80 total 20.01 last 128.05) - /unlink 3852 (time 1787752232.80 total 30.01 last 123.03) total: 4500 /unlink in 35.02 seconds: 128.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 643 (time 1787752252.64 total 10.01 last 64.25) - open/close 1318 (time 1787752262.65 total 20.02 last 67.41) - open/close 1971 (time 1787752272.66 total 30.03 last 65.25) - open/close 2678 (time 1787752282.68 total 40.04 last 70.59) - open/close 3392 (time 1787752292.70 total 50.06 last 71.26) - open/close 4050 (time 1787752302.70 total 60.06 last 65.79) total: 4500 open/close in 66.12 seconds: 68.06 ops/second - /unlink 1269 (time 1787752323.20 total 10.01 last 126.78) - /unlink 2454 (time 1787752333.20 total 20.01 last 118.44) - /unlink 3746 (time 1787752343.21 total 30.02 last 129.17) total: 4500 /unlink in 36.30 seconds: 123.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 721 (time 1787752363.91 total 10.01 last 72.04) - open/close 1345 (time 1787752373.91 total 20.01 last 62.40) - open/close 2030 (time 1787752383.91 total 30.02 last 68.45) - open/close 2675 (time 1787752393.92 total 40.02 last 64.49) - open/close 3382 (time 1787752403.92 total 50.02 last 70.66) - open/close 3928 (time 1787752413.94 total 60.04 last 54.50) total: 4500 open/close in 69.90 seconds: 64.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1070 (time 1787752439.30 total 10.01 last 106.92) - /unlink 2168 (time 1787752449.31 total 20.01 last 109.76) - /unlink 3302 (time 1787752459.31 total 30.01 last 113.37) - /unlink 4393 (time 1787752469.32 total 40.02 last 109.02) total: 4500 /unlink in 40.92 seconds: 109.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 715 (time 1787752486.55 total 10.01 last 71.46) - open/close 1349 (time 1787752496.56 total 20.02 last 63.31) - open/close 1987 (time 1787752506.58 total 30.03 last 63.71) - open/close 2637 (time 1787752516.58 total 40.03 last 64.99) - open/close 3344 (time 1787752526.59 total 50.05 last 70.61) - open/close 4050 (time 1787752536.59 total 60.05 last 70.56) total: 4500 open/close in 66.15 seconds: 68.03 ops/second - /unlink 1299 (time 1787752557.24 total 10.00 last 129.88) - /unlink 2438 (time 1787752567.25 total 20.02 last 113.74) - /unlink 3580 (time 1787752577.26 total 30.02 last 114.17) total: 4500 /unlink in 37.63 seconds: 119.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 654 (time 1787752600.07 total 10.01 last 65.32) - open/close 1308 (time 1787752610.08 total 20.02 last 65.32) - open/close 1859 (time 1787752620.11 total 30.06 last 54.93) - open/close 2433 (time 1787752630.13 total 40.07 last 57.33) - open/close 3072 (time 1787752640.13 total 50.07 last 63.86) - open/close 3621 (time 1787752650.14 total 60.08 last 54.86) - open/close 4223 (time 1787752660.15 total 70.09 last 60.13) total: 4500 open/close in 74.96 seconds: 60.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1173 (time 1787752679.74 total 10.01 last 117.20) - /unlink 2302 (time 1787752689.74 total 20.01 last 112.85) - /unlink 3244 (time 1787752699.74 total 30.02 last 94.17) - /unlink 4243 (time 1787752709.75 total 40.02 last 99.89) total: 4500 /unlink in 42.84 seconds: 105.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 626 (time 1787752728.46 total 10.00 last 62.58) - open/close 1269 (time 1787752738.47 total 20.00 last 64.29) - open/close 1861 (time 1787752748.47 total 30.01 last 59.15) - open/close 2468 (time 1787752758.50 total 40.04 last 60.51) - open/close 3084 (time 1787752768.51 total 50.05 last 61.57) - open/close 3687 (time 1787752778.52 total 60.05 last 60.26) - open/close 4305 (time 1787752788.52 total 70.06 last 61.78) total: 4500 open/close in 73.86 seconds: 60.93 ops/second - /unlink 1250 (time 1787752807.09 total 10.00 last 124.99) - /unlink 2468 (time 1787752817.10 total 20.01 last 121.72) - /unlink 3645 (time 1787752827.10 total 30.01 last 117.66) total: 4500 /unlink in 37.74 seconds: 119.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 700 (time 1787752850.30 total 10.01 last 69.96) - open/close 1256 (time 1787752860.30 total 20.01 last 55.59) - open/close 1832 (time 1787752870.33 total 30.03 last 57.45) - open/close 2431 (time 1787752880.33 total 40.04 last 59.87) - open/close 3049 (time 1787752890.34 total 50.04 last 61.76) - open/close 3730 (time 1787752900.35 total 60.05 last 68.07) - open/close 4365 (time 1787752910.36 total 70.06 last 63.43) total: 4500 open/close in 72.67 seconds: 61.92 ops/second - /unlink 1250 (time 1787752929.14 total 10.00 last 124.95) - /unlink 2543 (time 1787752939.14 total 20.00 last 129.29) - /unlink 3869 (time 1787752949.14 total 30.01 last 132.56) total: 4500 /unlink in 34.61 seconds: 130.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 712 (time 1787752969.26 total 10.01 last 71.13) - open/close 1389 (time 1787752979.27 total 20.02 last 67.63) - open/close 2064 (time 1787752989.27 total 30.02 last 67.47) - open/close 2612 (time 1787752999.29 total 40.04 last 54.72) - open/close 3229 (time 1787753009.31 total 50.07 last 61.53) - open/close 3850 (time 1787753019.31 total 60.07 last 62.09) total: 4500 open/close in 69.65 seconds: 64.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1297 (time 1787753043.51 total 10.00 last 129.69) - /unlink 2348 (time 1787753053.51 total 20.00 last 105.06) - /unlink 3344 (time 1787753063.52 total 30.01 last 99.52) total: 4500 /unlink in 39.80 seconds: 113.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 739 (time 1787753089.64 total 10.01 last 73.82) - open/close 1463 (time 1787753099.64 total 20.01 last 72.38) - open/close 2148 (time 1787753109.65 total 30.02 last 68.44) - open/close 2771 (time 1787753119.67 total 40.04 last 62.17) - open/close 3381 (time 1787753129.73 total 50.09 last 60.69) - open/close 3931 (time 1787753139.76 total 60.13 last 54.83) total: 4500 open/close in 68.06 seconds: 66.12 ops/second - /unlink 1325 (time 1787753162.10 total 10.00 last 132.49) - /unlink 2638 (time 1787753172.10 total 20.01 last 131.21) - /unlink 3983 (time 1787753182.10 total 30.01 last 134.47) total: 4500 /unlink in 33.98 seconds: 132.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 694 (time 1787753201.45 total 10.01 last 69.36) - open/close 1400 (time 1787753211.46 total 20.01 last 70.56) - open/close 2074 (time 1787753221.46 total 30.02 last 67.37) - open/close 2705 (time 1787753231.46 total 40.02 last 63.09) - open/close 3370 (time 1787753241.47 total 50.02 last 66.46) - open/close 4054 (time 1787753251.48 total 60.03 last 68.33) total: 4500 open/close in 65.42 seconds: 68.79 ops/second - /unlink 1319 (time 1787753271.87 total 10.01 last 131.74) - /unlink 2460 (time 1787753281.87 total 20.01 last 114.08) - /unlink 3823 (time 1787753291.87 total 30.01 last 136.28) total: 4500 /unlink in 35.00 seconds: 128.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 651 (time 1787753312.20 total 10.00 last 65.09) - open/close 1337 (time 1787753322.20 total 20.01 last 68.58) - open/close 1978 (time 1787753332.20 total 30.01 last 64.09) - open/close 2661 (time 1787753342.21 total 40.02 last 68.22) - open/close 3193 (time 1787753352.22 total 50.03 last 53.16) - open/close 3726 (time 1787753362.24 total 60.05 last 53.18) - open/close 4230 (time 1787753372.31 total 70.12 last 50.05) total: 4500 open/close in 75.40 seconds: 59.68 ops/second - /unlink 1259 (time 1787753392.98 total 10.00 last 125.86) - /unlink 2312 (time 1787753402.98 total 20.00 last 105.30) - /unlink 3389 (time 1787753412.99 total 30.01 last 107.61) - /unlink 4435 (time 1787753422.99 total 40.01 last 104.60) total: 4500 /unlink in 40.97 seconds: 109.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 564 (time 1787753438.47 total 10.04 last 56.19) - open/close 1208 (time 1787753448.52 total 20.09 last 64.08) - open/close 1854 (time 1787753458.54 total 30.10 last 64.51) - open/close 2500 (time 1787753468.54 total 40.11 last 64.57) - open/close 3145 (time 1787753478.54 total 50.11 last 64.48) - open/close 3829 (time 1787753488.55 total 60.11 last 68.38) - open/close 4453 (time 1787753498.56 total 70.13 last 62.32) total: 4500 open/close in 70.80 seconds: 63.56 ops/second - /unlink 1188 (time 1787753513.52 total 10.02 last 118.62) - /unlink 2364 (time 1787753523.52 total 20.02 last 117.59) - /unlink 3578 (time 1787753533.52 total 30.02 last 121.32) total: 4500 /unlink in 38.52 seconds: 116.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 696 (time 1787753557.12 total 10.00 last 69.58) - open/close 1338 (time 1787753567.13 total 20.01 last 64.17) - open/close 2011 (time 1787753577.13 total 30.01 last 67.26) - open/close 2707 (time 1787753587.15 total 40.02 last 69.52) - open/close 3302 (time 1787753597.16 total 50.03 last 59.44) - open/close 3877 (time 1787753607.17 total 60.05 last 57.39) - open/close 4477 (time 1787753617.17 total 70.05 last 60.00) total: 4500 open/close in 70.34 seconds: 63.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1410 (time 1787753632.13 total 10.00 last 140.99) - /unlink 2699 (time 1787753642.13 total 20.00 last 128.87) - /unlink 3930 (time 1787753652.14 total 30.01 last 123.06) total: 4500 /unlink in 34.26 seconds: 131.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 689 (time 1787753671.00 total 10.00 last 68.87) - open/close 1331 (time 1787753681.01 total 20.01 last 64.13) - open/close 2041 (time 1787753691.03 total 30.03 last 70.92) - open/close 2697 (time 1787753701.03 total 40.03 last 65.59) - open/close 3343 (time 1787753711.03 total 50.03 last 64.57) - open/close 3893 (time 1787753721.08 total 60.08 last 54.74) total: 4500 open/close in 69.56 seconds: 64.69 ops/second - /unlink 1305 (time 1787753745.92 total 10.01 last 130.42) - /unlink 2610 (time 1787753755.93 total 20.01 last 130.43) - /unlink 3717 (time 1787753765.93 total 30.01 last 110.69) total: 4500 /unlink in 36.10 seconds: 124.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 710 (time 1787753787.75 total 10.00 last 70.97) - open/close 1353 (time 1787753797.75 total 20.01 last 64.28) - open/close 2025 (time 1787753807.76 total 30.02 last 67.13) - open/close 2737 (time 1787753817.77 total 40.03 last 71.12) - open/close 3347 (time 1787753827.78 total 50.04 last 60.92) - open/close 3926 (time 1787753837.80 total 60.06 last 57.80) total: 4500 open/close in 69.89 seconds: 64.39 ops/second - /unlink 1354 (time 1787753861.89 total 10.00 last 135.35) - /unlink 2635 (time 1787753871.89 total 20.00 last 128.10) - /unlink 3940 (time 1787753881.89 total 30.01 last 130.46) total: 4500 /unlink in 33.58 seconds: 134.01 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95402 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: modules unloaded.