== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:25:21 (1763339121)
Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg255-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg255-server
oleg255-server: oleg255-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg255-server'
oleg255-server: oleg255-server.virtnet: executing load_modules_local
oleg255-server: Loading modules from /home/green/git/lustre-release/lustre
oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy
oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy
oleg255-server: detected 4 online CPUs by sysfs
oleg255-server: MODOPTS_LIBCFS=
oleg255-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg255-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg255-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg255-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all
pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg255-client.virtnet:  -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 645 (time 1763339296.19 total 10.01 last 64.43)
 - open/close 1372 (time 1763339306.19 total 20.01 last 72.69)
 - open/close 2213 (time 1763339316.20 total 30.03 last 83.98)
 - open/close 2988 (time 1763339326.21 total 40.03 last 77.46)
 - open/close 3720 (time 1763339336.21 total 50.03 last 73.18)
total: 4500 open/close in 59.80 seconds: 75.26 ops/second
 - /unlink 1460 (time 1763339359.46 total 10.00 last 145.99)
 - /unlink 2824 (time 1763339369.46 total 20.00 last 136.39)
 - /unlink 4234 (time 1763339379.46 total 30.00 last 140.98)
total: 4500 /unlink in 31.90 seconds: 141.09 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 639 (time 1763339395.71 total 10.01 last 63.85)
 - open/close 1433 (time 1763339405.72 total 20.01 last 79.36)
 - open/close 2202 (time 1763339415.75 total 30.05 last 76.65)
 - open/close 2963 (time 1763339425.76 total 40.05 last 76.04)
 - open/close 3766 (time 1763339435.76 total 50.06 last 80.25)
total: 4500 open/close in 59.62 seconds: 75.48 ops/second
 - /unlink 1222 (time 1763339459.28 total 10.01 last 122.12)
 - /unlink 2593 (time 1763339469.28 total 20.01 last 137.08)
 - /unlink 3992 (time 1763339479.28 total 30.01 last 139.86)
total: 4500 /unlink in 33.89 seconds: 132.80 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 795 (time 1763339498.00 total 10.00 last 79.46)
 - open/close 1625 (time 1763339508.00 total 20.00 last 83.00)
 - open/close 2463 (time 1763339518.01 total 30.01 last 83.74)
 - open/close 3258 (time 1763339528.01 total 40.02 last 79.47)
 - open/close 4034 (time 1763339538.02 total 50.02 last 77.55)
total: 4500 open/close in 56.02 seconds: 80.33 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1453 (time 1763339557.50 total 10.00 last 145.28)
 - /unlink 2819 (time 1763339567.50 total 20.00 last 136.57)
 - /unlink 4251 (time 1763339577.51 total 30.01 last 143.10)
total: 4500 /unlink in 31.88 seconds: 141.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 843 (time 1763339592.93 total 10.01 last 84.19)
 - open/close 1643 (time 1763339602.95 total 20.03 last 79.87)
 - open/close 2375 (time 1763339612.96 total 30.04 last 73.10)
 - open/close 3120 (time 1763339622.96 total 40.05 last 74.47)
 - open/close 3836 (time 1763339632.97 total 50.06 last 71.53)
total: 4500 open/close in 58.72 seconds: 76.64 ops/second
 - /unlink 1364 (time 1763339655.28 total 10.00 last 136.37)
 - /unlink 2603 (time 1763339665.29 total 20.01 last 123.75)
 - /unlink 3972 (time 1763339675.29 total 30.02 last 136.86)
total: 4500 /unlink in 34.42 seconds: 130.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 785 (time 1763339693.89 total 10.00 last 78.47)
 - open/close 1523 (time 1763339703.89 total 20.00 last 73.80)
 - open/close 2226 (time 1763339713.90 total 30.01 last 70.24)
 - open/close 2976 (time 1763339723.91 total 40.02 last 74.92)
 - open/close 3726 (time 1763339733.92 total 50.03 last 74.97)
 - open/close 4491 (time 1763339743.92 total 60.03 last 76.46)
total: 4500 open/close in 60.17 seconds: 74.79 ops/second
 - /unlink 1279 (time 1763339758.97 total 10.00 last 127.88)
 - /unlink 2581 (time 1763339768.98 total 20.01 last 130.13)
 - /unlink 4013 (time 1763339778.98 total 30.01 last 143.19)
total: 4500 /unlink in 33.58 seconds: 133.99 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 631 (time 1763339797.29 total 10.02 last 62.99)
 - open/close 1304 (time 1763339807.30 total 20.02 last 67.26)
 - open/close 1916 (time 1763339817.30 total 30.02 last 61.20)
 - open/close 2617 (time 1763339827.31 total 40.04 last 69.99)
 - open/close 3312 (time 1763339837.32 total 50.05 last 69.42)
 - open/close 4040 (time 1763339847.33 total 60.06 last 72.75)
total: 4500 open/close in 65.67 seconds: 68.53 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1383 (time 1763339866.67 total 10.00 last 138.27)
 - /unlink 2722 (time 1763339876.67 total 20.01 last 133.86)
 - /unlink 4123 (time 1763339886.67 total 30.01 last 140.06)
total: 4500 /unlink in 33.04 seconds: 136.20 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 595 (time 1763339904.43 total 10.01 last 59.46)
 - open/close 1270 (time 1763339914.45 total 20.02 last 67.39)
 - open/close 1890 (time 1763339924.48 total 30.05 last 61.83)
 - open/close 2340 (time 1763339934.49 total 40.06 last 44.94)
 - open/close 2978 (time 1763339944.51 total 50.08 last 63.71)
 - open/close 3679 (time 1763339954.51 total 60.09 last 70.05)
 - open/close 4365 (time 1763339964.52 total 70.10 last 68.53)
total: 4500 open/close in 71.74 seconds: 62.73 ops/second
 - /unlink 1182 (time 1763339981.51 total 10.00 last 118.15)
 - /unlink 2594 (time 1763339991.51 total 20.01 last 141.19)
 - /unlink 3975 (time 1763340001.51 total 30.01 last 138.06)
total: 4500 /unlink in 34.06 seconds: 132.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 764 (time 1763340019.99 total 10.01 last 76.29)
 - open/close 1562 (time 1763340029.99 total 20.02 last 79.79)
 - open/close 2373 (time 1763340039.99 total 30.02 last 81.07)
 - open/close 3149 (time 1763340050.00 total 40.03 last 77.51)
 - open/close 3903 (time 1763340060.01 total 50.04 last 75.35)
total: 4500 open/close in 57.15 seconds: 78.74 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1565 (time 1763340079.91 total 10.00 last 156.48)
 - /unlink 3070 (time 1763340089.91 total 20.01 last 150.37)
total: 4500 /unlink in 29.82 seconds: 150.89 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 812 (time 1763340113.34 total 10.01 last 81.13)
 - open/close 1605 (time 1763340123.35 total 20.02 last 79.22)
 - open/close 2405 (time 1763340133.35 total 30.03 last 79.95)
 - open/close 3228 (time 1763340143.37 total 40.04 last 82.18)
 - open/close 3959 (time 1763340153.37 total 50.04 last 73.08)
total: 4500 open/close in 56.86 seconds: 79.14 ops/second
 - /unlink 1453 (time 1763340173.68 total 10.00 last 145.27)
 - /unlink 2864 (time 1763340183.68 total 20.00 last 141.09)
 - /unlink 4266 (time 1763340193.69 total 30.01 last 140.13)
total: 4500 /unlink in 31.74 seconds: 141.78 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 695 (time 1763340209.39 total 10.01 last 69.45)
 - open/close 1516 (time 1763340219.41 total 20.03 last 81.94)
 - open/close 2309 (time 1763340229.43 total 30.04 last 79.17)
 - open/close 3133 (time 1763340239.43 total 40.04 last 82.40)
 - open/close 3953 (time 1763340249.43 total 50.05 last 81.95)
total: 4500 open/close in 56.44 seconds: 79.74 ops/second
 - /unlink 1394 (time 1763340270.37 total 10.00 last 139.37)
 - /unlink 2805 (time 1763340280.38 total 20.01 last 141.05)
 - /unlink 4161 (time 1763340290.38 total 30.01 last 135.58)
total: 4500 /unlink in 32.65 seconds: 137.81 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 777 (time 1763340307.02 total 10.03 last 77.49)
 - open/close 1582 (time 1763340317.03 total 20.03 last 80.49)
 - open/close 2380 (time 1763340327.03 total 30.03 last 79.76)
 - open/close 3210 (time 1763340337.03 total 40.03 last 82.99)
 - open/close 3969 (time 1763340347.03 total 50.04 last 75.88)
total: 4500 open/close in 56.51 seconds: 79.64 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1359 (time 1763340367.53 total 10.01 last 135.78)
 - /unlink 2806 (time 1763340377.54 total 20.01 last 144.62)
 - /unlink 4341 (time 1763340387.54 total 30.01 last 153.50)
total: 4500 /unlink in 31.22 seconds: 144.15 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 772 (time 1763340402.47 total 10.02 last 77.03)
 - open/close 1648 (time 1763340412.48 total 20.04 last 87.46)
 - open/close 2393 (time 1763340422.49 total 30.05 last 74.41)
 - open/close 3183 (time 1763340432.50 total 40.05 last 78.97)
 - open/close 4013 (time 1763340442.50 total 50.05 last 82.99)
total: 4500 open/close in 55.75 seconds: 80.72 ops/second
 - /unlink 1356 (time 1763340462.01 total 10.01 last 135.46)
 - /unlink 2761 (time 1763340472.01 total 20.01 last 140.47)
 - /unlink 4007 (time 1763340482.02 total 30.02 last 124.54)
total: 4500 /unlink in 33.15 seconds: 135.75 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 837 (time 1763340499.47 total 10.01 last 83.66)
 - open/close 1656 (time 1763340509.48 total 20.01 last 81.84)
 - open/close 2415 (time 1763340519.49 total 30.02 last 75.80)
 - open/close 3142 (time 1763340529.52 total 40.05 last 72.54)
 - open/close 3958 (time 1763340539.52 total 50.05 last 81.60)
total: 4500 open/close in 57.55 seconds: 78.19 ops/second
 - /unlink 1533 (time 1763340560.98 total 10.00 last 153.27)
 - /unlink 2917 (time 1763340570.98 total 20.00 last 138.37)
 - /unlink 4108 (time 1763340580.98 total 30.01 last 119.09)
total: 4500 /unlink in 33.47 seconds: 134.43 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 840 (time 1763340599.74 total 10.00 last 83.99)
 - open/close 1618 (time 1763340609.75 total 20.00 last 77.77)
 - open/close 2493 (time 1763340619.75 total 30.01 last 87.49)
 - open/close 3295 (time 1763340629.75 total 40.01 last 80.17)
 - open/close 4148 (time 1763340639.75 total 50.01 last 85.29)
total: 4500 open/close in 54.22 seconds: 82.99 ops/second
 - /unlink 1538 (time 1763340657.71 total 10.00 last 153.77)
 - /unlink 2926 (time 1763340667.72 total 20.01 last 138.69)
 - /unlink 4425 (time 1763340677.72 total 30.01 last 149.86)
total: 4500 /unlink in 30.73 seconds: 146.44 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 825 (time 1763340692.37 total 10.01 last 82.43)
 - open/close 1660 (time 1763340702.38 total 20.01 last 83.49)
 - open/close 2559 (time 1763340712.38 total 30.02 last 89.84)
 - open/close 3388 (time 1763340722.39 total 40.02 last 82.86)
 - open/close 4186 (time 1763340732.39 total 50.02 last 79.80)
total: 4500 open/close in 53.78 seconds: 83.67 ops/second
 - /unlink 1376 (time 1763340749.30 total 10.01 last 137.51)
 - /unlink 2794 (time 1763340759.30 total 20.01 last 141.79)
 - /unlink 4233 (time 1763340769.30 total 30.02 last 143.78)
total: 4500 /unlink in 31.95 seconds: 140.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 772 (time 1763340784.90 total 10.00 last 77.19)
 - open/close 1470 (time 1763340794.91 total 20.01 last 69.72)
 - open/close 2191 (time 1763340804.91 total 30.01 last 72.10)
 - open/close 3021 (time 1763340814.92 total 40.02 last 82.98)
 - open/close 3788 (time 1763340824.92 total 50.02 last 76.66)
total: 4500 open/close in 58.47 seconds: 76.97 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1555 (time 1763340845.97 total 10.01 last 155.30)
 - /unlink 3005 (time 1763340855.97 total 20.02 last 144.96)
total: 4500 /unlink in 29.87 seconds: 150.67 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 875 (time 1763340879.80 total 10.01 last 87.38)
 - open/close 1579 (time 1763340889.80 total 20.02 last 70.38)
 - open/close 2385 (time 1763340899.82 total 30.03 last 80.46)
 - open/close 3186 (time 1763340909.83 total 40.05 last 80.01)
 - open/close 3986 (time 1763340919.83 total 50.05 last 79.95)
total: 4500 open/close in 57.29 seconds: 78.54 ops/second
 - /unlink 1585 (time 1763340942.14 total 10.01 last 158.39)
 - /unlink 3065 (time 1763340952.14 total 20.01 last 147.98)
total: 4500 /unlink in 29.90 seconds: 150.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 810 (time 1763340976.01 total 10.01 last 80.94)
 - open/close 1689 (time 1763340986.01 total 20.01 last 87.85)
 - open/close 2482 (time 1763340996.02 total 30.02 last 79.26)
 - open/close 3250 (time 1763341006.02 total 40.02 last 76.77)
 - open/close 4070 (time 1763341016.03 total 50.03 last 81.98)
total: 4500 open/close in 54.73 seconds: 82.22 ops/second
 - /unlink 1467 (time 1763341033.84 total 10.01 last 146.62)
 - /unlink 2867 (time 1763341043.84 total 20.01 last 139.99)
 - /unlink 4336 (time 1763341053.85 total 30.01 last 146.86)
total: 4500 /unlink in 31.28 seconds: 143.86 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93877 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg255-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg255-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server
stop mds service on oleg255-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server
stop mds service on oleg255-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg255-server'
oleg255-server: oleg255-server.virtnet: executing unload_modules_local
oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.