== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:26:35 (1786692395) Stopping clients: oleg405-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg405-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg405-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg405-server oleg405-server: oleg405-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg405-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg405-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 642 (time 1786692618.52 total 10.02 last 64.07) - open/close 1179 (time 1786692628.53 total 20.03 last 53.64) - open/close 1806 (time 1786692638.54 total 30.04 last 62.64) - open/close 2540 (time 1786692648.56 total 40.06 last 73.30) - open/close 3317 (time 1786692658.56 total 50.06 last 77.69) - open/close 4054 (time 1786692668.57 total 60.07 last 73.61) total: 4500 open/close in 66.29 seconds: 67.88 ops/second - /unlink 1335 (time 1786692689.57 total 10.00 last 133.46) - /unlink 2530 (time 1786692699.57 total 20.01 last 119.46) - /unlink 3706 (time 1786692709.58 total 30.02 last 117.44) total: 4500 /unlink in 34.94 seconds: 128.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 568 (time 1786692730.64 total 10.00 last 56.79) - open/close 1159 (time 1786692740.66 total 20.02 last 59.02) - open/close 1827 (time 1786692750.66 total 30.02 last 66.79) - open/close 2541 (time 1786692760.69 total 40.04 last 71.21) - open/close 3212 (time 1786692770.70 total 50.06 last 67.02) - open/close 3941 (time 1786692780.71 total 60.07 last 72.81) total: 4500 open/close in 69.27 seconds: 64.97 ops/second - /unlink 1179 (time 1786692805.22 total 10.00 last 117.89) - /unlink 2345 (time 1786692815.23 total 20.00 last 116.57) - /unlink 3570 (time 1786692825.23 total 30.00 last 122.49) total: 4500 /unlink in 37.98 seconds: 118.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 561 (time 1786692849.52 total 10.01 last 56.02) - open/close 1063 (time 1786692859.52 total 20.01 last 50.20) - open/close 1529 (time 1786692869.53 total 30.03 last 46.51) - open/close 2086 (time 1786692879.55 total 40.04 last 55.64) - open/close 2829 (time 1786692889.55 total 50.05 last 74.27) - open/close 3438 (time 1786692899.55 total 60.05 last 60.88) - open/close 4036 (time 1786692909.56 total 70.06 last 59.75) total: 4500 open/close in 76.77 seconds: 58.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1188 (time 1786692931.85 total 10.00 last 118.79) - /unlink 2534 (time 1786692941.85 total 20.00 last 134.58) - /unlink 3725 (time 1786692951.87 total 30.02 last 118.90) total: 4500 /unlink in 35.89 seconds: 125.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 559 (time 1786692973.74 total 10.04 last 55.67) - open/close 1199 (time 1786692983.75 total 20.05 last 63.96) - open/close 1767 (time 1786692993.76 total 30.06 last 56.75) - open/close 2557 (time 1786693003.76 total 40.06 last 78.93) - open/close 3256 (time 1786693013.77 total 50.07 last 69.85) - open/close 3698 (time 1786693023.78 total 60.08 last 44.17) - open/close 4038 (time 1786693033.78 total 70.08 last 33.99) total: 4500 open/close in 77.99 seconds: 57.70 ops/second - /unlink 905 (time 1786693057.47 total 10.00 last 90.48) - /unlink 2029 (time 1786693067.48 total 20.01 last 112.31) - /unlink 3173 (time 1786693077.48 total 30.01 last 114.37) - /unlink 4401 (time 1786693087.49 total 40.02 last 122.72) total: 4500 /unlink in 40.93 seconds: 109.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 683 (time 1786693103.63 total 10.01 last 68.24) - open/close 1343 (time 1786693113.63 total 20.01 last 66.00) - open/close 2037 (time 1786693123.63 total 30.01 last 69.40) - open/close 2741 (time 1786693133.64 total 40.01 last 70.37) - open/close 3596 (time 1786693143.65 total 50.02 last 85.44) - open/close 4351 (time 1786693153.65 total 60.03 last 75.45) total: 4500 open/close in 62.92 seconds: 71.52 ops/second - /unlink 937 (time 1786693172.91 total 10.00 last 93.67) - /unlink 2157 (time 1786693182.91 total 20.01 last 121.97) - /unlink 3638 (time 1786693192.91 total 30.01 last 148.05) total: 4500 /unlink in 35.84 seconds: 125.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 783 (time 1786693213.73 total 10.01 last 78.19) - open/close 1540 (time 1786693223.74 total 20.02 last 75.63) - open/close 2076 (time 1786693233.77 total 30.05 last 53.45) - open/close 2617 (time 1786693243.79 total 40.07 last 54.01) - open/close 3439 (time 1786693253.79 total 50.07 last 82.17) - open/close 4335 (time 1786693263.79 total 60.07 last 89.58) total: 4500 open/close in 62.33 seconds: 72.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1477 (time 1786693280.44 total 10.00 last 147.67) - /unlink 2793 (time 1786693290.44 total 20.00 last 131.59) - /unlink 4173 (time 1786693300.45 total 30.01 last 137.87) total: 4500 /unlink in 32.78 seconds: 137.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 472 (time 1786693320.35 total 10.00 last 47.19) - open/close 986 (time 1786693330.36 total 20.01 last 51.35) - open/close 1509 (time 1786693340.38 total 30.03 last 52.19) - open/close 2189 (time 1786693350.38 total 40.04 last 67.96) - open/close 2996 (time 1786693360.40 total 50.06 last 80.54) - open/close 3720 (time 1786693370.41 total 60.07 last 72.34) - open/close 4448 (time 1786693380.41 total 70.07 last 72.77) total: 4500 open/close in 70.59 seconds: 63.75 ops/second - /unlink 1414 (time 1786693395.07 total 10.00 last 141.34) - /unlink 2747 (time 1786693405.07 total 20.01 last 133.29) - /unlink 4166 (time 1786693415.08 total 30.02 last 141.76) total: 4500 /unlink in 32.66 seconds: 137.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 718 (time 1786693433.57 total 10.00 last 71.79) - open/close 1658 (time 1786693443.57 total 20.01 last 93.97) - open/close 2342 (time 1786693453.58 total 30.01 last 68.38) - open/close 3054 (time 1786693463.58 total 40.01 last 71.16) - open/close 3823 (time 1786693473.59 total 50.02 last 76.84) - open/close 4483 (time 1786693483.59 total 60.02 last 66.00) total: 4500 open/close in 60.20 seconds: 74.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 748 (time 1786693498.75 total 10.00 last 74.79) - /unlink 1490 (time 1786693508.76 total 20.01 last 74.13) - /unlink 2283 (time 1786693518.76 total 30.01 last 79.29) - /unlink 3316 (time 1786693528.77 total 40.02 last 103.20) - /unlink 4432 (time 1786693538.77 total 50.03 last 111.54) total: 4500 /unlink in 50.70 seconds: 88.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 677 (time 1786693555.86 total 10.00 last 67.70) - open/close 1293 (time 1786693565.87 total 20.01 last 61.53) - open/close 2011 (time 1786693575.87 total 30.01 last 71.79) - open/close 2753 (time 1786693585.88 total 40.03 last 74.10) - open/close 3542 (time 1786693595.90 total 50.04 last 78.79) - open/close 4272 (time 1786693605.91 total 60.05 last 72.93) total: 4500 open/close in 62.84 seconds: 71.61 ops/second - /unlink 1459 (time 1786693623.14 total 10.00 last 145.89) - /unlink 2962 (time 1786693633.14 total 20.00 last 150.30) - /unlink 4367 (time 1786693643.14 total 30.00 last 140.44) total: 4500 /unlink in 31.26 seconds: 143.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 634 (time 1786693659.66 total 10.06 last 63.05) - open/close 1270 (time 1786693669.68 total 20.07 last 63.49) - open/close 2047 (time 1786693679.71 total 30.10 last 77.46) - open/close 2677 (time 1786693689.72 total 40.12 last 62.90) - open/close 3371 (time 1786693699.74 total 50.14 last 69.27) - open/close 4116 (time 1786693709.78 total 60.18 last 74.23) total: 4500 open/close in 64.63 seconds: 69.62 ops/second - /unlink 1420 (time 1786693728.56 total 10.00 last 141.98) - /unlink 2870 (time 1786693738.56 total 20.00 last 145.00) - /unlink 4169 (time 1786693748.56 total 30.01 last 129.85) total: 4500 /unlink in 32.30 seconds: 139.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 814 (time 1786693765.98 total 10.01 last 81.36) - open/close 1549 (time 1786693775.98 total 20.01 last 73.47) - open/close 2317 (time 1786693786.00 total 30.02 last 76.69) - open/close 3160 (time 1786693796.02 total 40.04 last 84.16) - open/close 3966 (time 1786693806.02 total 50.04 last 80.59) total: 4500 open/close in 57.02 seconds: 78.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1491 (time 1786693827.15 total 10.00 last 149.07) - /unlink 2975 (time 1786693837.15 total 20.01 last 148.34) - /unlink 4403 (time 1786693847.17 total 30.02 last 142.62) total: 4500 /unlink in 30.89 seconds: 145.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 716 (time 1786693863.16 total 10.00 last 71.58) - open/close 1278 (time 1786693873.19 total 20.04 last 56.00) - open/close 2106 (time 1786693883.20 total 30.05 last 82.72) - open/close 2883 (time 1786693893.21 total 40.05 last 77.68) - open/close 3612 (time 1786693903.21 total 50.06 last 72.85) - open/close 4274 (time 1786693913.22 total 60.07 last 66.14) total: 4500 open/close in 63.57 seconds: 70.79 ops/second - /unlink 1277 (time 1786693931.36 total 10.00 last 127.68) - /unlink 2761 (time 1786693941.37 total 20.00 last 148.35) - /unlink 4258 (time 1786693951.37 total 30.01 last 149.58) total: 4500 /unlink in 31.69 seconds: 142.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 866 (time 1786693967.63 total 10.00 last 86.56) - open/close 1539 (time 1786693977.64 total 20.02 last 67.19) - open/close 2258 (time 1786693987.65 total 30.03 last 71.83) - open/close 2950 (time 1786693997.66 total 40.04 last 69.16) - open/close 3717 (time 1786694007.67 total 50.04 last 76.64) total: 4500 open/close in 59.98 seconds: 75.02 ops/second - /unlink 1461 (time 1786694031.65 total 10.00 last 146.03) - /unlink 2783 (time 1786694041.65 total 20.01 last 132.18) - /unlink 4219 (time 1786694051.66 total 30.01 last 143.55) total: 4500 /unlink in 32.12 seconds: 140.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 799 (time 1786694068.98 total 10.00 last 79.86) - open/close 1514 (time 1786694078.98 total 20.01 last 71.49) - open/close 2291 (time 1786694088.99 total 30.01 last 77.65) - open/close 3126 (time 1786694098.99 total 40.02 last 83.44) - open/close 3904 (time 1786694109.00 total 50.03 last 77.75) total: 4500 open/close in 57.88 seconds: 77.74 ops/second - /unlink 1476 (time 1786694131.58 total 10.01 last 147.50) - /unlink 2603 (time 1786694141.59 total 20.01 last 112.67) - /unlink 3938 (time 1786694151.59 total 30.01 last 133.46) total: 4500 /unlink in 34.12 seconds: 131.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 863 (time 1786694170.80 total 10.01 last 86.23) - open/close 1689 (time 1786694180.81 total 20.02 last 82.52) - open/close 2453 (time 1786694190.82 total 30.03 last 76.32) - open/close 3266 (time 1786694200.82 total 40.04 last 81.24) - open/close 4004 (time 1786694210.83 total 50.04 last 73.78) total: 4500 open/close in 56.15 seconds: 80.14 ops/second - /unlink 1393 (time 1786694232.15 total 10.01 last 139.18) - /unlink 2791 (time 1786694242.15 total 20.01 last 139.74) - /unlink 4254 (time 1786694252.15 total 30.02 last 146.26) total: 4500 /unlink in 31.95 seconds: 140.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 821 (time 1786694269.28 total 10.00 last 82.08) - open/close 1555 (time 1786694279.29 total 20.01 last 73.32) - open/close 2282 (time 1786694289.29 total 30.02 last 72.68) - open/close 3004 (time 1786694299.30 total 40.02 last 72.14) - open/close 3651 (time 1786694309.30 total 50.03 last 64.66) - open/close 4494 (time 1786694319.30 total 60.03 last 84.30) total: 4500 open/close in 60.08 seconds: 74.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1253 (time 1786694333.91 total 10.00 last 125.28) - /unlink 2424 (time 1786694343.92 total 20.01 last 117.04) - /unlink 3756 (time 1786694353.93 total 30.02 last 133.04) total: 4500 /unlink in 35.67 seconds: 126.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 718 (time 1786694374.38 total 10.00 last 71.77) - open/close 1417 (time 1786694384.38 total 20.01 last 69.89) - open/close 1988 (time 1786694394.39 total 30.01 last 57.06) - open/close 2586 (time 1786694404.40 total 40.02 last 59.74) - open/close 3148 (time 1786694414.41 total 50.03 last 56.17) - open/close 3785 (time 1786694424.43 total 60.05 last 63.58) - open/close 4460 (time 1786694434.43 total 70.05 last 67.49) total: 4500 open/close in 70.85 seconds: 63.52 ops/second - /unlink 1066 (time 1786694450.73 total 10.01 last 106.51) - /unlink 2336 (time 1786694460.73 total 20.01 last 126.98) - /unlink 3698 (time 1786694470.74 total 30.03 last 135.97) total: 4500 /unlink in 36.25 seconds: 124.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 796 (time 1786694492.57 total 10.02 last 79.44) - open/close 1643 (time 1786694502.59 total 20.04 last 84.54) - open/close 2436 (time 1786694512.59 total 30.04 last 79.27) - open/close 3044 (time 1786694522.60 total 40.05 last 60.77) - open/close 3703 (time 1786694532.60 total 50.05 last 65.88) - open/close 4405 (time 1786694542.61 total 60.06 last 70.17) total: 4500 open/close in 61.55 seconds: 73.11 ops/second - /unlink 1255 (time 1786694559.48 total 10.00 last 125.48) - /unlink 2523 (time 1786694569.48 total 20.00 last 126.77) - /unlink 3803 (time 1786694579.48 total 30.01 last 127.95) total: 4500 /unlink in 36.62 seconds: 122.88 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95189 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: modules unloaded.