|
2018-05-24
ยง
|
| 19:30 |
<twentyafterfour@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 19:14 |
<twentyafterfour> |
Starting the MediaWiki train for Thursday May 24, today I will be deploying wmf/1.32.0-wmf.5 to all wikis |
[production] |
| 18:31 |
<mutante> |
netmon1002, netmon2001: systemctl mask uwsgi; systemctl reset-failed - to fix Icinga alert about broken DPKG since last netbox deploy and to match existing status on labtestweb2001 |
[production] |
| 18:21 |
<XioNoX> |
repool esams |
[production] |
| 18:01 |
<no_justification> |
gerrit restarting on cobalt, back soon |
[production] |
| 17:50 |
<mutante> |
mwdebug1001 - apt-get remove to clean up packages in "non ii" states |
[production] |
| 16:59 |
<anomie> |
Running populateExternallinksIndex60.php on group 1 for T59176 |
[production] |
| 16:50 |
<XioNoX> |
depooled esams - investigating issues |
[production] |
| 16:31 |
<anomie> |
Running deduplicateArchiveRevId.php on group 2 for T193180 |
[production] |
| 16:28 |
<jynus> |
manually failover the backup host for m2 to db1117:3322 |
[production] |
| 16:22 |
<mutante> |
tin apt-get clean saved 7% disk space on / - fixing disk space alert |
[production] |
| 16:19 |
<marostegui> |
Deploy schema change on db1099:3311 - T191519 T188299 T190148 |
[production] |
| 16:19 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 01m 13s) |
[production] |
| 16:16 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 01m 17s) |
[production] |
| 16:15 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase/repo/maintenance/dispatchChanges.php: [[gerrit:434940|Dont use WikibaseRepo class in dispatchChanges constructor]] (duration: 10m 28s) |
[production] |
| 16:13 |
<mutante> |
deploy2001 - let mwdeploy own .~tmp~ in /srv/mediawiki-staging |
[production] |
| 15:48 |
<cmjohnson> |
swapped failed disk 0 db1065 |
[production] |
| 15:06 |
<moritzm> |
installing glibc updates from stretch point release |
[production] |
| 14:57 |
<marostegui> |
Deploy schema change on dbstore1001:s1 - T191519 T188299 T190148 |
[production] |
| 14:56 |
<papaul> |
shutting down elastic2020 for maintenance |
[production] |
| 14:51 |
<moritzm> |
upgrading mwdebug servers in eqiad to wikidiff 1.7.0 |
[production] |
| 14:09 |
<akosiaris> |
empty ganeti2004 for stretch reimage |
[production] |
| 14:00 |
<mutante> |
deploy2001: scap pull to sync. then add as scap master and host (gerrit:433616) |
[production] |
| 13:37 |
<akosiaris> |
rebalance row_A codfw ganeti nodegroup. Fully upgrade to stretch now |
[production] |
| 13:33 |
<marostegui> |
Deploy schema change on db1067 - T191519 T188299 T190148 |
[production] |
| 13:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 01m 00s) |
[production] |
| 13:27 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 56s) |
[production] |
| 13:27 |
<anomie> |
Running deduplicateArchiveRevId.php on group 1 for T193180 |
[production] |
| 13:25 |
<moritzm> |
upgrading mw1262-mw1265 (canary hosts) to wikidiff 1.7.0 |
[production] |
| 13:13 |
<moritzm> |
upgrading mw1261 (canary host) to wikidiff 1.7.0 |
[production] |
| 11:31 |
<moritzm> |
rebooting mw1261, mw1276, mw1319, mw1312, mw1258, mw1221 to use Intel microcode updates |
[production] |
| 11:19 |
<marostegui> |
Deploy schema change on db1080 - T191519 T188299 T190148 |
[production] |
| 11:18 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 08s) |
[production] |
| 11:10 |
<marostegui> |
Deploy schema change on dbstore1002:s1 - T191519 T188299 T190148 |
[production] |
| 11:05 |
<XioNoX> |
GTT work on eqiad-esams link starting soon |
[production] |
| 10:55 |
<krinkle@tin> |
Synchronized php-1.32.0-wmf.4/includes/resourceloader/ResourceLoaderUserModule.php: T195380 (duration: 01m 08s) |
[production] |
| 10:53 |
<Krinkle> |
Unexpected dirty git status at tin:/srv/mediawiki-staging/php-1.32.0-wmf.4/extensions/JADE (1 file is locally deleted, but not committed) |
[production] |
| 10:51 |
<krinkle@tin> |
Synchronized php-1.32.0-wmf.5/includes/resourceloader/ResourceLoaderUserModule.php: T195380 (duration: 01m 08s) |
[production] |
| 09:55 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 2/2 - T190327 (duration: 01m 06s) |
[production] |
| 09:55 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327 (duration: 00m 42s) |
[production] |
| 09:54 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327 |
[production] |
| 09:53 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2, take #2 - T190327 (duration: 01m 08s) |
[production] |
| 09:37 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2 - T190327 (duration: 01m 09s) |
[production] |
| 09:33 |
<pnorman@tin> |
Finished deploy [kartotherian/deploy@9fc09ef]: Do a fresh deploy of Kartotherian to production (duration: 02m 35s) |
[production] |
| 09:31 |
<pnorman@tin> |
Started deploy [kartotherian/deploy@9fc09ef]: Do a fresh deploy of Kartotherian to production |
[production] |
| 09:27 |
<pnorman@tin> |
Finished deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 (duration: 00m 22s) |
[production] |
| 09:27 |
<pnorman@tin> |
Started deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 |
[production] |
| 09:26 |
<pnorman@tin> |
Finished deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 (duration: 00m 23s) |
[production] |
| 09:25 |
<pnorman@tin> |
Started deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 |
[production] |
| 09:25 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@f66dacb]: Correctly commit offsets for multi-topic rules (duration: 00m 49s) |
[production] |