== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:24:30 (1753442670) Stopping clients: oleg326-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg326-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg326-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg326-server oleg326-server: oleg326-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg326-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg326-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg326-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre oleg326-client: fail_loc=0x1312 oleg326-client: fail_val=5 oleg326-server: fail_loc=0x1312 oleg326-server: fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 2425 (time 1753442777.84 total 10.00 last 242.39) total: 4500 open/close in 18.88 seconds: 238.35 ops/second total: 4500 /unlink in 10.00 seconds: 450.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 1846 (time 1753442810.04 total 10.00 last 184.57) - open/close 3904 (time 1753442820.04 total 20.00 last 205.76) total: 4500 open/close in 22.44 seconds: 200.53 ops/second total: 4500 /unlink in 9.94 seconds: 452.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 2772 (time 1753442845.61 total 10.00 last 277.13) total: 4500 open/close in 16.89 seconds: 266.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 total: 4500 /unlink in 7.29 seconds: 617.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 2762 (time 1753442872.63 total 10.01 last 276.04) total: 4500 open/close in 17.58 seconds: 256.04 ops/second - /unlink 4010 (time 1753442891.90 total 10.00 last 400.98) total: 4500 /unlink in 11.21 seconds: 401.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 2279 (time 1753442904.66 total 10.00 last 227.83) - open/close 4297 (time 1753442914.66 total 20.00 last 201.80) total: 4500 open/close in 20.88 seconds: 215.50 ops/second - /unlink 4341 (time 1753442927.13 total 10.00 last 434.06) total: 4500 /unlink in 10.32 seconds: 436.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 2217 (time 1753442939.12 total 10.00 last 221.66) total: 4500 open/close in 18.93 seconds: 237.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 4152 (time 1753442959.79 total 10.00 last 415.17) total: 4500 /unlink in 10.58 seconds: 425.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 2068 (time 1753442971.94 total 10.00 last 206.79) - open/close 3994 (time 1753442981.95 total 20.00 last 192.53) total: 4500 open/close in 22.08 seconds: 203.83 ops/second total: 4500 /unlink in 9.84 seconds: 457.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1985 (time 1753443006.87 total 10.00 last 198.44) - open/close 4323 (time 1753443016.87 total 20.00 last 233.78) total: 4500 open/close in 21.01 seconds: 214.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 total: 4500 /unlink in 9.17 seconds: 490.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 2321 (time 1753443040.14 total 10.00 last 232.04) - open/close 3950 (time 1753443050.14 total 20.01 last 162.85) total: 4500 open/close in 23.02 seconds: 195.46 ops/second - /unlink 2688 (time 1753443065.01 total 10.01 last 268.66) - /unlink 4476 (time 1753443075.01 total 20.01 last 178.77) total: 4500 /unlink in 20.13 seconds: 223.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 765 (time 1753443089.55 total 10.00 last 76.46) - open/close 1706 (time 1753443099.56 total 20.01 last 94.07) - open/close 2609 (time 1753443109.56 total 30.01 last 90.24) - open/close 3554 (time 1753443119.57 total 40.02 last 94.45) total: 4500 open/close in 47.92 seconds: 93.90 ops/second - /unlink 2508 (time 1753443139.57 total 10.00 last 250.72) total: 4500 /unlink in 18.18 seconds: 247.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1114 (time 1753443159.99 total 10.00 last 111.40) - open/close 2821 (time 1753443169.99 total 20.00 last 170.64) - open/close 4343 (time 1753443180.00 total 30.01 last 152.14) total: 4500 open/close in 31.23 seconds: 144.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2470 (time 1753443193.41 total 10.00 last 246.98) total: 4500 /unlink in 16.83 seconds: 267.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1070 (time 1753443211.91 total 10.01 last 106.90) - open/close 2436 (time 1753443221.91 total 20.01 last 136.54) - open/close 3796 (time 1753443231.92 total 30.02 last 135.92) total: 4500 open/close in 35.89 seconds: 125.37 ops/second - /unlink 2193 (time 1753443249.92 total 10.00 last 219.22) - /unlink 4270 (time 1753443259.92 total 20.00 last 207.68) total: 4500 /unlink in 21.25 seconds: 211.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1076 (time 1753443273.47 total 10.00 last 107.55) - open/close 2184 (time 1753443283.47 total 20.01 last 110.77) - open/close 3340 (time 1753443293.48 total 30.01 last 115.54) - open/close 4441 (time 1753443303.48 total 40.02 last 110.07) total: 4500 open/close in 40.56 seconds: 110.96 ops/second - /unlink 2032 (time 1753443316.38 total 10.00 last 203.17) total: 4500 /unlink in 18.57 seconds: 242.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1270 (time 1753443337.50 total 10.00 last 126.99) - open/close 2304 (time 1753443347.50 total 20.00 last 103.38) - open/close 3356 (time 1753443357.51 total 30.01 last 105.15) - open/close 4467 (time 1753443367.52 total 40.02 last 111.00) total: 4500 open/close in 40.30 seconds: 111.65 ops/second - /unlink 2206 (time 1753443380.00 total 10.00 last 220.54) total: 4500 /unlink in 18.63 seconds: 241.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 925 (time 1753443400.74 total 10.01 last 92.44) - open/close 1507 (time 1753443410.76 total 20.02 last 58.09) - open/close 2147 (time 1753443420.77 total 30.03 last 63.94) - open/close 2678 (time 1753443430.81 total 40.08 last 52.88) - open/close 3237 (time 1753443440.81 total 50.08 last 55.90) - open/close 3780 (time 1753443450.84 total 60.11 last 54.14) - open/close 4293 (time 1753443460.85 total 70.11 last 51.29) total: 4500 open/close in 73.28 seconds: 61.41 ops/second - /unlink 1460 (time 1753443478.84 total 10.01 last 145.86) - /unlink 2926 (time 1753443488.85 total 20.01 last 146.55) - /unlink 4404 (time 1753443498.85 total 30.02 last 147.77) total: 4500 /unlink in 30.64 seconds: 146.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 821 (time 1753443513.72 total 10.00 last 82.09) - open/close 1643 (time 1753443523.72 total 20.01 last 82.13) - open/close 2535 (time 1753443533.73 total 30.02 last 89.14) - open/close 3380 (time 1753443543.74 total 40.03 last 84.38) - open/close 4233 (time 1753443553.75 total 50.04 last 85.21) total: 4500 open/close in 53.27 seconds: 84.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1487 (time 1753443570.70 total 10.00 last 148.68) - /unlink 3114 (time 1753443580.70 total 20.00 last 162.66) total: 4500 /unlink in 29.12 seconds: 154.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 879 (time 1753443602.99 total 10.00 last 87.89) - open/close 1778 (time 1753443612.99 total 20.01 last 89.84) - open/close 2641 (time 1753443622.99 total 30.01 last 86.29) - open/close 3553 (time 1753443633.00 total 40.02 last 91.13) - open/close 4366 (time 1753443643.02 total 50.03 last 81.17) total: 4500 open/close in 51.46 seconds: 87.45 ops/second - /unlink 1640 (time 1753443658.68 total 10.00 last 163.96) - /unlink 3127 (time 1753443668.68 total 20.01 last 148.64) total: 4500 /unlink in 29.49 seconds: 152.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 879 (time 1753443692.04 total 10.00 last 87.89) - open/close 1794 (time 1753443702.05 total 20.01 last 91.41) - open/close 2705 (time 1753443712.05 total 30.02 last 91.05) - open/close 3584 (time 1753443722.06 total 40.02 last 87.84) total: 4500 open/close in 49.61 seconds: 90.70 ops/second - /unlink 1831 (time 1753443744.24 total 10.00 last 183.09) - /unlink 3338 (time 1753443754.24 total 20.00 last 150.69) total: 4500 /unlink in 27.27 seconds: 164.99 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11149: 93328 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg326-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local oleg326-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg326-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.