== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:01:51 (1782237711)
Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f)
Stopping client oleg219-client.virtnet /mnt/lustre opts:-f
Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f)
Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server
Stopping /mnt/lustre-ost2 (opts:-f) on oleg219-server
oleg219-server: oleg219-server.virtnet: executing set_hostid
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
loading modules on: 'oleg219-server'
oleg219-server: oleg219-server.virtnet: executing load_modules_local
oleg219-server: Loading modules from /home/green/git/lustre-release/lustre
oleg219-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg219-server: detected 4 online CPUs by sysfs
oleg219-server: MODOPTS_LIBCFS=
oleg219-server: Force libcfs to create 2 CPU partitions
Formatting mgs, mds, osts
Format mds1: /dev/mapper/mds1_flakey
Format mds2: /dev/mapper/mds2_flakey
Format ost1: /dev/mapper/ost1_flakey
Format ost2: /dev/mapper/ost2_flakey
start mds service on oleg219-server
Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds1_flakey
Started lustre-MDT0000
start mds service on oleg219-server
Start mds2: mount -t lustre -o localrecov  /dev/mapper/mds2_flakey /mnt/lustre-mds2
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/mds2_flakey
Started lustre-MDT0001
oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg219-server
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Commit the device label on /dev/mapper/ost1_flakey
Started lustre-OST0000
oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg219-client.virtnet:  -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre
fail_loc=0x1312
fail_val=5
fail_loc=0x1312
fail_val=5
striped dir -i0 -c1 -H all_char  /mnt/lustre/d135.conf-sanity
seq.cli-lustre-OST0000-super.width=0x1ffffff
mdd.lustre-MDT0000.changelog_mask=ALL
mdd.lustre-MDT0001.changelog_mask=ALL
mdd.lustre-MDT0000.changelog_mask=+hsm
mdd.lustre-MDT0001.changelog_mask=+hsm
Registered 2 changelog users: 'cl1 cl1'

Wrap arround changelog catalog
lustre-MDT0000: clear the changelog for cl1 to record #12998
 - open/close 695 (time 1782237938.62 total 10.01 last 69.40)
 - open/close 1409 (time 1782237948.62 total 20.01 last 71.40)
 - open/close 2203 (time 1782237958.63 total 30.02 last 79.34)
 - open/close 2975 (time 1782237968.64 total 40.03 last 77.16)
 - open/close 3784 (time 1782237978.65 total 50.04 last 80.81)
 - open/close 4471 (time 1782237988.65 total 60.04 last 68.68)
total: 4500 open/close in 60.39 seconds: 74.52 ops/second
 - /unlink 1483 (time 1782238003.89 total 10.00 last 148.29)
 - /unlink 3052 (time 1782238013.90 total 20.00 last 156.83)
total: 4500 /unlink in 29.44 seconds: 152.83 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #25998
 - open/close 807 (time 1782238037.77 total 10.01 last 80.63)
 - open/close 1554 (time 1782238047.79 total 20.02 last 74.58)
 - open/close 2316 (time 1782238057.79 total 30.03 last 76.18)
 - open/close 3011 (time 1782238067.80 total 40.03 last 69.47)
 - open/close 3640 (time 1782238077.80 total 50.04 last 62.85)
 - open/close 4361 (time 1782238087.82 total 60.06 last 71.97)
total: 4500 open/close in 62.29 seconds: 72.24 ops/second
 - /unlink 1363 (time 1782238104.94 total 10.01 last 136.22)
 - /unlink 2889 (time 1782238114.94 total 20.01 last 152.59)
 - /unlink 4400 (time 1782238124.95 total 30.02 last 150.95)
total: 4500 /unlink in 30.80 seconds: 146.12 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #38998
 - open/close 861 (time 1782238139.58 total 10.01 last 86.04)
 - open/close 1613 (time 1782238149.58 total 20.01 last 75.19)
 - open/close 2294 (time 1782238159.59 total 30.02 last 68.03)
 - open/close 2960 (time 1782238169.60 total 40.03 last 66.53)
 - open/close 3722 (time 1782238179.62 total 50.05 last 76.07)
 - open/close 4381 (time 1782238189.63 total 60.05 last 65.85)
total: 4500 open/close in 61.77 seconds: 72.85 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #51998
 - /unlink 1200 (time 1782238206.13 total 10.00 last 119.95)
 - /unlink 2570 (time 1782238216.14 total 20.01 last 136.92)
 - /unlink 3941 (time 1782238226.14 total 30.01 last 137.08)
total: 4500 /unlink in 33.72 seconds: 133.45 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #64998
 - open/close 815 (time 1782238244.96 total 10.01 last 81.42)
 - open/close 1590 (time 1782238254.97 total 20.02 last 77.43)
 - open/close 2322 (time 1782238264.98 total 30.03 last 73.14)
 - open/close 3025 (time 1782238274.98 total 40.03 last 70.29)
 - open/close 3783 (time 1782238284.99 total 50.03 last 75.75)
total: 4500 open/close in 58.82 seconds: 76.51 ops/second
 - /unlink 1549 (time 1782238308.70 total 10.00 last 154.83)
 - /unlink 2676 (time 1782238318.70 total 20.01 last 112.64)
 - /unlink 3648 (time 1782238328.71 total 30.02 last 97.08)
total: 4500 /unlink in 36.98 seconds: 121.68 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #77998
 - open/close 675 (time 1782238351.13 total 10.01 last 67.42)
 - open/close 1498 (time 1782238361.15 total 20.02 last 82.20)
 - open/close 2293 (time 1782238371.15 total 30.02 last 79.50)
 - open/close 3114 (time 1782238381.15 total 40.03 last 82.05)
 - open/close 3803 (time 1782238391.17 total 50.05 last 68.77)
 - open/close 4470 (time 1782238401.18 total 60.06 last 66.65)
total: 4500 open/close in 60.30 seconds: 74.63 ops/second
 - /unlink 1437 (time 1782238416.07 total 10.00 last 143.70)
 - /unlink 2708 (time 1782238426.07 total 20.00 last 127.08)
 - /unlink 4200 (time 1782238436.07 total 30.00 last 149.16)
total: 4500 /unlink in 32.14 seconds: 140.00 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #90998
 - open/close 773 (time 1782238453.07 total 10.01 last 77.26)
 - open/close 1469 (time 1782238463.07 total 20.01 last 69.59)
 - open/close 2122 (time 1782238473.07 total 30.01 last 65.29)
 - open/close 2836 (time 1782238483.08 total 40.01 last 71.39)
 - open/close 3540 (time 1782238493.08 total 50.01 last 70.37)
 - open/close 4194 (time 1782238503.08 total 60.02 last 65.38)
total: 4500 open/close in 64.53 seconds: 69.73 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #103998
 - /unlink 1210 (time 1782238522.40 total 10.02 last 120.75)
 - /unlink 2433 (time 1782238532.41 total 20.03 last 122.23)
 - /unlink 3626 (time 1782238542.42 total 30.04 last 119.20)
total: 4500 /unlink in 38.33 seconds: 117.40 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #116998
 - open/close 595 (time 1782238566.68 total 10.01 last 59.42)
 - open/close 1125 (time 1782238576.69 total 20.02 last 52.96)
 - open/close 1595 (time 1782238586.71 total 30.04 last 46.93)
 - open/close 2309 (time 1782238596.71 total 40.04 last 71.35)
 - open/close 3079 (time 1782238606.71 total 50.04 last 77.00)
 - open/close 3781 (time 1782238616.73 total 60.06 last 70.08)
 - open/close 4422 (time 1782238626.73 total 70.06 last 64.08)
total: 4500 open/close in 71.47 seconds: 62.97 ops/second
 - /unlink 1311 (time 1782238643.15 total 10.00 last 131.07)
 - /unlink 2599 (time 1782238653.15 total 20.00 last 128.77)
 - /unlink 3923 (time 1782238663.16 total 30.01 last 132.37)
total: 4500 /unlink in 34.31 seconds: 131.16 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #129998
 - open/close 689 (time 1782238682.49 total 10.02 last 68.77)
 - open/close 1260 (time 1782238692.49 total 20.02 last 57.09)
 - open/close 1946 (time 1782238702.51 total 30.04 last 68.45)
 - open/close 2706 (time 1782238712.52 total 40.05 last 75.95)
 - open/close 3366 (time 1782238722.52 total 50.05 last 65.97)
 - open/close 4104 (time 1782238732.53 total 60.06 last 73.78)
total: 4500 open/close in 64.71 seconds: 69.54 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #142998
 - /unlink 1247 (time 1782238751.51 total 10.00 last 124.66)
 - /unlink 2259 (time 1782238761.51 total 20.01 last 101.18)
 - /unlink 3430 (time 1782238771.51 total 30.01 last 117.07)
total: 4500 /unlink in 38.48 seconds: 116.93 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #155998
 - open/close 844 (time 1782238795.07 total 10.00 last 84.36)
 - open/close 1531 (time 1782238805.07 total 20.00 last 68.70)
 - open/close 2182 (time 1782238815.08 total 30.01 last 65.04)
 - open/close 2932 (time 1782238825.08 total 40.02 last 74.98)
 - open/close 3751 (time 1782238835.09 total 50.02 last 81.85)
total: 4500 open/close in 59.58 seconds: 75.53 ops/second
 - /unlink 1308 (time 1782238858.98 total 10.00 last 130.74)
 - /unlink 2566 (time 1782238868.98 total 20.01 last 125.76)
 - /unlink 3844 (time 1782238878.99 total 30.02 last 127.69)
total: 4500 /unlink in 35.87 seconds: 125.47 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #168998
 - open/close 808 (time 1782238900.36 total 10.03 last 80.60)
 - open/close 1679 (time 1782238910.37 total 20.04 last 87.00)
 - open/close 2502 (time 1782238920.38 total 30.04 last 82.23)
 - open/close 3207 (time 1782238930.42 total 40.08 last 70.26)
 - open/close 3869 (time 1782238940.43 total 50.09 last 66.13)
total: 4500 open/close in 58.76 seconds: 76.58 ops/second
 - /unlink 1119 (time 1782238963.39 total 10.01 last 111.84)
 - /unlink 2235 (time 1782238973.39 total 20.01 last 111.58)
 - /unlink 3361 (time 1782238983.39 total 30.01 last 112.54)
 - /unlink 4367 (time 1782238993.39 total 40.01 last 100.58)
total: 4500 /unlink in 41.87 seconds: 107.49 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #181998
 - open/close 441 (time 1782239012.91 total 10.00 last 44.09)
 - open/close 1064 (time 1782239022.93 total 20.02 last 62.15)
 - open/close 1740 (time 1782239032.95 total 30.04 last 67.47)
 - open/close 2473 (time 1782239042.97 total 40.06 last 73.19)
 - open/close 3222 (time 1782239052.99 total 50.08 last 74.74)
 - open/close 3843 (time 1782239062.99 total 60.08 last 62.08)
total: 4500 open/close in 69.15 seconds: 65.07 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #194998
 - /unlink 1429 (time 1782239087.28 total 10.00 last 142.85)
 - /unlink 2775 (time 1782239097.28 total 20.00 last 134.59)
 - /unlink 4048 (time 1782239107.29 total 30.02 last 127.15)
total: 4500 /unlink in 33.82 seconds: 133.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #207998
 - open/close 589 (time 1782239127.13 total 10.01 last 58.83)
 - open/close 1249 (time 1782239137.13 total 20.02 last 65.97)
 - open/close 1919 (time 1782239147.14 total 30.02 last 66.96)
 - open/close 2548 (time 1782239157.15 total 40.03 last 62.87)
 - open/close 3036 (time 1782239167.15 total 50.04 last 48.75)
 - open/close 3509 (time 1782239177.17 total 60.05 last 47.22)
 - open/close 4052 (time 1782239187.19 total 70.07 last 54.18)
total: 4500 open/close in 77.01 seconds: 58.44 ops/second
 - /unlink 1047 (time 1782239209.52 total 10.00 last 104.66)
 - /unlink 2093 (time 1782239219.52 total 20.00 last 104.60)
 - /unlink 3102 (time 1782239229.53 total 30.01 last 100.85)
 - /unlink 4214 (time 1782239239.53 total 40.01 last 111.20)
total: 4500 /unlink in 42.09 seconds: 106.92 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #220998
 - open/close 675 (time 1782239256.06 total 10.00 last 67.50)
 - open/close 1308 (time 1782239266.07 total 20.00 last 63.28)
 - open/close 1902 (time 1782239276.07 total 30.01 last 59.38)
 - open/close 2630 (time 1782239286.08 total 40.01 last 72.74)
 - open/close 3308 (time 1782239296.08 total 50.02 last 67.78)
 - open/close 4051 (time 1782239306.08 total 60.02 last 74.28)
total: 4500 open/close in 65.76 seconds: 68.43 ops/second
 - /unlink 1368 (time 1782239326.32 total 10.00 last 136.79)
 - /unlink 2557 (time 1782239336.32 total 20.00 last 118.86)
 - /unlink 3706 (time 1782239346.32 total 30.00 last 114.89)
total: 4500 /unlink in 36.13 seconds: 124.55 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #233998
lustre-MDT0000: clear the changelog for cl1 to record #246998
 - open/close 697 (time 1782239368.33 total 10.01 last 69.64)
 - open/close 1438 (time 1782239378.33 total 20.01 last 74.08)
 - open/close 2259 (time 1782239388.34 total 30.02 last 82.04)
 - open/close 2794 (time 1782239398.34 total 40.03 last 53.46)
 - open/close 3481 (time 1782239408.36 total 50.04 last 68.62)
 - open/close 4144 (time 1782239418.36 total 60.04 last 66.26)
total: 4500 open/close in 65.45 seconds: 68.75 ops/second
 - /unlink 1453 (time 1782239439.37 total 10.00 last 145.25)
 - /unlink 2859 (time 1782239449.37 total 20.01 last 140.57)
 - /unlink 4201 (time 1782239459.38 total 30.01 last 134.11)
total: 4500 /unlink in 32.24 seconds: 139.59 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #259998
 - open/close 686 (time 1782239477.01 total 10.02 last 68.49)
 - open/close 1425 (time 1782239487.02 total 20.03 last 73.83)
 - open/close 2159 (time 1782239497.03 total 30.03 last 73.37)
 - open/close 2988 (time 1782239507.03 total 40.03 last 82.88)
 - open/close 3798 (time 1782239517.04 total 50.04 last 80.93)
total: 4500 open/close in 59.38 seconds: 75.78 ops/second
 - /unlink 1280 (time 1782239541.86 total 10.00 last 127.94)
 - /unlink 2430 (time 1782239551.86 total 20.01 last 114.92)
 - /unlink 3822 (time 1782239561.87 total 30.02 last 139.15)
total: 4500 /unlink in 34.88 seconds: 129.02 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #272998
 - open/close 833 (time 1782239581.91 total 10.00 last 83.27)
 - open/close 1531 (time 1782239591.91 total 20.01 last 69.79)
 - open/close 2117 (time 1782239601.93 total 30.02 last 58.53)
 - open/close 2750 (time 1782239611.94 total 40.03 last 63.23)
 - open/close 3459 (time 1782239621.94 total 50.03 last 70.87)
 - open/close 4262 (time 1782239631.94 total 60.04 last 80.28)
total: 4500 open/close in 63.04 seconds: 71.39 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #285998
 - /unlink 1302 (time 1782239650.08 total 10.01 last 130.10)
 - /unlink 2574 (time 1782239660.09 total 20.02 last 127.02)
 - /unlink 3678 (time 1782239670.10 total 30.03 last 110.35)
total: 4500 /unlink in 37.48 seconds: 120.06 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #298998
 - open/close 622 (time 1782239694.61 total 10.01 last 62.13)
 - open/close 1253 (time 1782239704.61 total 20.01 last 63.09)
 - open/close 2059 (time 1782239714.64 total 30.04 last 80.36)
 - open/close 2816 (time 1782239724.65 total 40.05 last 75.66)
 - open/close 3478 (time 1782239734.65 total 50.05 last 66.18)
 - open/close 4159 (time 1782239744.65 total 60.05 last 68.09)
total: 4500 open/close in 64.27 seconds: 70.02 ops/second
 - /unlink 1183 (time 1782239763.52 total 10.00 last 118.28)
 - /unlink 2371 (time 1782239773.52 total 20.00 last 118.78)
 - /unlink 3690 (time 1782239783.52 total 30.01 last 131.86)
total: 4500 /unlink in 35.82 seconds: 125.62 ops/second
lustre-MDT0000: clear the changelog for cl1 to record #311998
 - open/close 714 (time 1782239804.04 total 10.02 last 71.29)
 - open/close 1398 (time 1782239814.05 total 20.02 last 68.36)
 - open/close 2119 (time 1782239824.06 total 30.03 last 72.02)
 - open/close 2875 (time 1782239834.06 total 40.03 last 75.59)
 - open/close 3489 (time 1782239844.08 total 50.05 last 61.31)
 - open/close 4173 (time 1782239854.08 total 60.05 last 68.40)
total: 4500 open/close in 65.57 seconds: 68.63 ops/second
 - /unlink 1388 (time 1782239874.81 total 10.01 last 138.63)
 - /unlink 2725 (time 1782239884.82 total 20.02 last 133.61)
 - /unlink 4160 (time 1782239894.83 total 30.03 last 143.36)
total: 4500 /unlink in 32.84 seconds: 137.04 ops/second
kill changelog reader
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95070 Terminated              coproc COPROC $LFS changelog --follow $service  (wd: ~)
lustre-MDT0001: clear the changelog for cl1 of all records
lustre-MDT0001: Deregistered changelog user #1
lustre-MDT0000: clear the changelog for cl1 of all records
lustre-MDT0000: Deregistered changelog user #1

Cleanup test_135
umount lustre on /mnt/lustre.....
Stopping client oleg219-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg219-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server
stop mds service on oleg219-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server
stop mds service on oleg219-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg219-server

unloading modules via unload_modules_local on: 'oleg219-server'
oleg219-server: oleg219-server.virtnet: executing unload_modules_local
oleg219-server: 
modules unloaded.