== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:36:34 (1787848594) Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg223-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg223-server oleg223-server: oleg223-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg223-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg223-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 754 (time 1787848826.63 total 10.00 last 75.37) - open/close 1510 (time 1787848836.64 total 20.02 last 75.50) - open/close 2230 (time 1787848846.64 total 30.02 last 71.99) - open/close 2991 (time 1787848856.65 total 40.02 last 76.06) - open/close 3618 (time 1787848866.66 total 50.03 last 62.64) - open/close 4290 (time 1787848876.67 total 60.05 last 67.13) total: 4500 open/close in 62.93 seconds: 71.51 ops/second - /unlink 1498 (time 1787848894.57 total 10.00 last 149.77) - /unlink 2803 (time 1787848904.58 total 20.01 last 130.46) - /unlink 3998 (time 1787848914.58 total 30.01 last 119.47) total: 4500 /unlink in 33.81 seconds: 133.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 738 (time 1787848933.74 total 10.01 last 73.76) - open/close 1328 (time 1787848943.76 total 20.02 last 58.90) - open/close 1918 (time 1787848953.76 total 30.02 last 59.00) - open/close 2520 (time 1787848963.77 total 40.03 last 60.14) - open/close 3010 (time 1787848973.77 total 50.04 last 48.99) - open/close 3612 (time 1787848983.79 total 60.05 last 60.12) - open/close 4231 (time 1787848993.79 total 70.05 last 61.86) total: 4500 open/close in 74.42 seconds: 60.47 ops/second - /unlink 1184 (time 1787849014.14 total 10.02 last 118.19) - /unlink 2372 (time 1787849024.14 total 20.02 last 118.78) - /unlink 3549 (time 1787849034.14 total 30.02 last 117.69) total: 4500 /unlink in 39.48 seconds: 113.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 689 (time 1787849059.55 total 10.02 last 68.79) - open/close 1329 (time 1787849069.56 total 20.03 last 63.90) - open/close 1982 (time 1787849079.56 total 30.03 last 65.29) - open/close 2726 (time 1787849089.58 total 40.05 last 74.30) - open/close 3318 (time 1787849099.58 total 50.05 last 59.19) - open/close 3831 (time 1787849109.59 total 60.06 last 51.22) - open/close 4360 (time 1787849119.61 total 70.08 last 52.84) total: 4500 open/close in 73.09 seconds: 61.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1225 (time 1787849139.93 total 10.02 last 122.26) - /unlink 2463 (time 1787849149.93 total 20.03 last 123.73) - /unlink 3593 (time 1787849159.95 total 30.05 last 112.75) total: 4500 /unlink in 37.79 seconds: 119.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 669 (time 1787849183.90 total 10.00 last 66.88) - open/close 1215 (time 1787849193.90 total 20.01 last 54.55) - open/close 1852 (time 1787849203.92 total 30.02 last 63.62) - open/close 2495 (time 1787849213.92 total 40.03 last 64.26) - open/close 3098 (time 1787849223.94 total 50.05 last 60.20) - open/close 3759 (time 1787849233.94 total 60.05 last 66.07) - open/close 4444 (time 1787849243.95 total 70.05 last 68.49) total: 4500 open/close in 70.94 seconds: 63.43 ops/second - /unlink 1230 (time 1787849261.12 total 10.02 last 122.81) - /unlink 2567 (time 1787849271.13 total 20.02 last 133.65) - /unlink 3750 (time 1787849281.13 total 30.02 last 118.27) total: 4500 /unlink in 38.36 seconds: 117.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 613 (time 1787849305.48 total 10.02 last 61.18) - open/close 1231 (time 1787849315.48 total 20.02 last 61.79) - open/close 1956 (time 1787849325.49 total 30.03 last 72.42) - open/close 2653 (time 1787849335.50 total 40.04 last 69.66) - open/close 3383 (time 1787849345.50 total 50.04 last 72.95) - open/close 4012 (time 1787849355.51 total 60.05 last 62.85) total: 4500 open/close in 66.87 seconds: 67.30 ops/second - /unlink 1175 (time 1787849377.87 total 10.00 last 117.50) - /unlink 2227 (time 1787849387.87 total 20.00 last 105.16) - /unlink 3279 (time 1787849397.88 total 30.02 last 105.06) total: 4500 /unlink in 38.93 seconds: 115.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 651 (time 1787849422.45 total 10.02 last 64.97) - open/close 1384 (time 1787849432.48 total 20.05 last 73.09) - open/close 2028 (time 1787849442.48 total 30.05 last 64.37) - open/close 2527 (time 1787849452.48 total 40.06 last 49.88) - open/close 3212 (time 1787849462.49 total 50.07 last 68.45) - open/close 3930 (time 1787849472.50 total 60.07 last 71.73) total: 4500 open/close in 69.50 seconds: 64.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 973 (time 1787849498.18 total 10.01 last 97.19) - /unlink 1707 (time 1787849508.19 total 20.02 last 73.32) - /unlink 2742 (time 1787849518.19 total 30.03 last 103.47) - /unlink 3729 (time 1787849528.20 total 40.03 last 98.68) total: 4500 /unlink in 47.23 seconds: 95.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 563 (time 1787849552.35 total 10.00 last 56.28) - open/close 1151 (time 1787849562.35 total 20.00 last 58.79) - open/close 1761 (time 1787849572.36 total 30.02 last 60.93) - open/close 2310 (time 1787849582.36 total 40.02 last 54.89) - open/close 3030 (time 1787849592.39 total 50.05 last 71.78) - open/close 3761 (time 1787849602.39 total 60.05 last 73.08) total: 4500 open/close in 68.63 seconds: 65.57 ops/second - /unlink 1551 (time 1787849625.83 total 10.01 last 154.91) - /unlink 2723 (time 1787849635.83 total 20.01 last 117.18) - /unlink 4048 (time 1787849645.83 total 30.01 last 132.49) total: 4500 /unlink in 33.05 seconds: 136.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 518 (time 1787849665.04 total 10.01 last 51.77) - open/close 1193 (time 1787849675.05 total 20.02 last 67.43) - open/close 1720 (time 1787849685.06 total 30.03 last 52.65) - open/close 2358 (time 1787849695.07 total 40.04 last 63.71) - open/close 3159 (time 1787849705.07 total 50.04 last 80.09) - open/close 3791 (time 1787849715.08 total 60.04 last 63.18) - open/close 4468 (time 1787849725.09 total 70.06 last 67.59) total: 4500 open/close in 70.36 seconds: 63.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1458 (time 1787849740.37 total 10.00 last 145.77) - /unlink 2621 (time 1787849750.37 total 20.01 last 116.23) - /unlink 3960 (time 1787849760.37 total 30.01 last 133.87) total: 4500 /unlink in 34.42 seconds: 130.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 746 (time 1787849780.42 total 10.00 last 74.58) - open/close 1434 (time 1787849790.44 total 20.02 last 68.67) - open/close 2042 (time 1787849800.46 total 30.04 last 60.66) - open/close 2669 (time 1787849810.46 total 40.05 last 62.68) - open/close 3270 (time 1787849820.49 total 50.07 last 59.94) - open/close 3920 (time 1787849830.50 total 60.08 last 64.94) total: 4500 open/close in 67.93 seconds: 66.25 ops/second - /unlink 1248 (time 1787849852.99 total 10.01 last 124.67) - /unlink 2386 (time 1787849863.00 total 20.02 last 113.68) - /unlink 3944 (time 1787849873.00 total 30.03 last 155.73) total: 4500 /unlink in 33.90 seconds: 132.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 699 (time 1787849892.42 total 10.03 last 69.72) - open/close 1380 (time 1787849902.42 total 20.03 last 68.06) - open/close 2105 (time 1787849912.43 total 30.04 last 72.41) - open/close 2878 (time 1787849922.45 total 40.06 last 77.14) - open/close 3708 (time 1787849932.45 total 50.06 last 83.00) - open/close 4425 (time 1787849942.47 total 60.08 last 71.60) total: 4500 open/close in 61.21 seconds: 73.52 ops/second - /unlink 1239 (time 1787849958.83 total 10.01 last 123.73) - /unlink 2468 (time 1787849968.85 total 20.03 last 122.73) - /unlink 3715 (time 1787849978.86 total 30.03 last 124.61) total: 4500 /unlink in 35.54 seconds: 126.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 612 (time 1787850001.01 total 10.02 last 61.05) - open/close 1178 (time 1787850011.01 total 20.03 last 56.57) - open/close 1732 (time 1787850021.02 total 30.03 last 55.39) - open/close 2232 (time 1787850031.03 total 40.05 last 49.92) - open/close 2896 (time 1787850041.03 total 50.05 last 66.38) - open/close 3531 (time 1787850051.04 total 60.06 last 63.43) - open/close 4366 (time 1787850061.05 total 70.06 last 83.50) total: 4500 open/close in 72.26 seconds: 62.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1220 (time 1787850078.31 total 10.00 last 121.99) - /unlink 2527 (time 1787850088.31 total 20.01 last 130.62) - /unlink 3791 (time 1787850098.32 total 30.01 last 126.31) total: 4500 /unlink in 35.25 seconds: 127.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 639 (time 1787850118.65 total 10.01 last 63.86) - open/close 1303 (time 1787850128.65 total 20.01 last 66.37) - open/close 1918 (time 1787850138.68 total 30.03 last 61.35) - open/close 2534 (time 1787850148.69 total 40.05 last 61.49) - open/close 3387 (time 1787850158.70 total 50.06 last 85.26) - open/close 3948 (time 1787850168.71 total 60.07 last 56.05) total: 4500 open/close in 67.25 seconds: 66.92 ops/second - /unlink 1214 (time 1787850190.76 total 10.00 last 121.40) - /unlink 2507 (time 1787850200.76 total 20.00 last 129.28) - /unlink 4030 (time 1787850210.76 total 30.00 last 152.25) total: 4500 /unlink in 33.38 seconds: 134.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 763 (time 1787850228.99 total 10.02 last 76.16) - open/close 1335 (time 1787850239.00 total 20.02 last 57.16) - open/close 1858 (time 1787850249.00 total 30.03 last 52.28) - open/close 2411 (time 1787850259.01 total 40.04 last 55.24) - open/close 3040 (time 1787850269.02 total 50.04 last 62.87) - open/close 3659 (time 1787850279.03 total 60.05 last 61.82) - open/close 4323 (time 1787850289.05 total 70.07 last 66.27) total: 4500 open/close in 72.99 seconds: 61.65 ops/second - /unlink 1305 (time 1787850306.47 total 10.00 last 130.48) - /unlink 2637 (time 1787850316.47 total 20.01 last 133.15) - /unlink 3656 (time 1787850326.48 total 30.02 last 101.77) total: 4500 /unlink in 38.58 seconds: 116.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 543 (time 1787850352.07 total 10.00 last 54.29) - open/close 1166 (time 1787850362.08 total 20.01 last 62.23) - open/close 1988 (time 1787850372.08 total 30.01 last 82.19) - open/close 2763 (time 1787850382.11 total 40.04 last 77.29) - open/close 3342 (time 1787850392.11 total 50.04 last 57.89) - open/close 3874 (time 1787850402.11 total 60.05 last 53.18) - open/close 4411 (time 1787850412.14 total 70.07 last 53.55) total: 4500 open/close in 71.42 seconds: 63.01 ops/second - /unlink 1337 (time 1787850428.79 total 10.01 last 133.52) - /unlink 2512 (time 1787850438.80 total 20.02 last 117.47) - /unlink 3839 (time 1787850448.80 total 30.02 last 132.69) total: 4500 /unlink in 34.79 seconds: 129.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 733 (time 1787850469.83 total 10.00 last 73.30) - open/close 1444 (time 1787850479.84 total 20.01 last 71.04) - open/close 2243 (time 1787850489.84 total 30.02 last 79.84) - open/close 3062 (time 1787850499.85 total 40.02 last 81.83) - open/close 3785 (time 1787850509.87 total 50.04 last 72.19) - open/close 4399 (time 1787850519.87 total 60.04 last 61.38) total: 4500 open/close in 61.50 seconds: 73.17 ops/second - /unlink 1274 (time 1787850536.83 total 10.00 last 127.35) - /unlink 2363 (time 1787850546.83 total 20.01 last 108.84) - /unlink 3540 (time 1787850556.83 total 30.01 last 117.70) total: 4500 /unlink in 38.17 seconds: 117.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 692 (time 1787850581.43 total 10.00 last 69.18) - open/close 1353 (time 1787850591.43 total 20.00 last 66.10) - open/close 2019 (time 1787850601.44 total 30.02 last 66.50) - open/close 2665 (time 1787850611.46 total 40.04 last 64.48) - open/close 3262 (time 1787850621.48 total 50.05 last 59.62) - open/close 3910 (time 1787850631.49 total 60.07 last 64.70) total: 4500 open/close in 69.40 seconds: 64.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1221 (time 1787850656.42 total 10.00 last 122.07) - /unlink 2777 (time 1787850666.42 total 20.00 last 155.56) - /unlink 4366 (time 1787850676.43 total 30.02 last 158.71) total: 4500 /unlink in 31.66 seconds: 142.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 697 (time 1787850693.43 total 10.01 last 69.64) - open/close 1245 (time 1787850703.44 total 20.02 last 54.76) - open/close 1946 (time 1787850713.45 total 30.02 last 70.05) - open/close 2564 (time 1787850723.45 total 40.03 last 61.76) - open/close 3185 (time 1787850733.45 total 50.03 last 62.10) - open/close 3711 (time 1787850743.46 total 60.03 last 52.59) - open/close 4364 (time 1787850753.46 total 70.03 last 65.30) total: 4500 open/close in 72.44 seconds: 62.12 ops/second - /unlink 1487 (time 1787850771.36 total 10.01 last 148.59) - /unlink 2819 (time 1787850781.36 total 20.01 last 133.14) - /unlink 4349 (time 1787850791.36 total 30.02 last 152.93) total: 4500 /unlink in 31.29 seconds: 143.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 697 (time 1787850808.04 total 10.00 last 69.67) - open/close 1527 (time 1787850818.06 total 20.02 last 82.84) - open/close 2222 (time 1787850828.06 total 30.03 last 69.47) - open/close 2675 (time 1787850838.10 total 40.06 last 45.15) - open/close 3248 (time 1787850848.12 total 50.08 last 57.17) - open/close 3785 (time 1787850858.13 total 60.10 last 53.61) total: 4500 open/close in 69.81 seconds: 64.46 ops/second - /unlink 1439 (time 1787850883.15 total 10.00 last 143.87) - /unlink 2787 (time 1787850893.15 total 20.00 last 134.78) - /unlink 4349 (time 1787850903.16 total 30.01 last 156.05) total: 4500 /unlink in 31.34 seconds: 143.58 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95379 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: modules unloaded.