|
2017-07-19
§
|
| 09:58 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1065 - T166204 (duration: 00m 47s) |
[production] |
| 09:16 |
<ema> |
finish up codfw cache_text/upload varnish/kernel upgrades |
[production] |
| 09:05 |
<oblivian@neodymium> |
conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=(restbase-async|citoid) |
[production] |
| 09:03 |
<XioNoX> |
codfw repooled in dns - T170380 |
[production] |
| 09:01 |
<hashar> |
restarting nodepool for upgrade 0.1.1-wmf7 -> 0.1.1-wmf8 |
[production] |
| 08:48 |
<moritzm> |
uploaded nodepool 0.1.1+wmf8 to apt.wikimedia.org |
[production] |
| 08:29 |
<XioNoX> |
asw-c-codfw back online - T170380 |
[production] |
| 08:28 |
<XioNoX> |
asw-c-codfw restarted 8min ago for switch upgrade - T170380 |
[production] |
| 08:09 |
<akosiaris> |
disable librenms crons on netmon1002 for a while |
[production] |
| 07:57 |
<marostegui> |
Drop migrateuser_medium from s7 - T170310 |
[production] |
| 07:49 |
<oblivian@neodymium> |
conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=(restbase-async|citoid) |
[production] |
| 05:26 |
<_joe_> |
ran systemctl reset-failed on codfw jobrunners after the jobrunner process was activated by mistake running scap at 21.20 UTC yesterday |
[production] |
| 03:03 |
<l10nupdate@tin> |
LocalisationUpdate failed: git pull of extensions failed |
[production] |
| 01:27 |
<mutante> |
netmon1001 - stopping all the services, killing snmpwalk, disarming keyholder |
[production] |
| 00:35 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Remove rcs1001 and rcs1002 from CommonSettings wgRCFeeds. Stops a load of logspam T170157 (duration: 00m 48s) |
[production] |
|
2017-07-18
§
|
| 23:53 |
<mutante> |
netmon1002 - copied Letsencrypt cert/key for librenms from netmon1001 for migration after netmon1002 has been reinstalled and now has RAID. (T159756) |
[production] |
| 23:40 |
<thcipriani@tin> |
Synchronized wmf-config/InterwikiSortOrders.php: SWAT: [[gerrit:365451|Add din to InterwikiSortOrders]] T168518 (duration: 00m 46s) |
[production] |
| 23:35 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:365942|Add Welsh mobile logo (just changes 'k' to 'c']] PART II (duration: 00m 46s) |
[production] |
| 23:34 |
<thcipriani@tin> |
Synchronized static/images/mobile/copyright/wikipedia-wordmark-cy.svg: SWAT: [[gerrit:365942|Add Welsh mobile logo (just changes 'k' to 'c']] PART I (duration: 00m 47s) |
[production] |
| 23:27 |
<thcipriani@tin> |
Synchronized php-1.30.0-wmf.9/extensions/Thanks/extension.json: SWAT: [[gerrit:366168|Add missing jQueryMsg dependency for mobile diff view]] T170917 (duration: 00m 47s) |
[production] |
| 23:22 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:360371|Enable OOjs UI EditPage buttons on all Wikipedias]] T162849 (duration: 00m 47s) |
[production] |
| 23:13 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:365884|Enable CodeMirror on simplewiki for better testing and more exposure]] (duration: 00m 48s) |
[production] |
| 22:58 |
<thcipriani> |
restared jobrunner on mw1299.eqiad.wmnet mw1168.eqiad.wmnet mw1164.eqiad.wmnet mw1305.eqiad.wmnet mw1304.eqiad.wmnet mw1301.eqiad.wmnet mw1259.eqiad.wmnet mw1166.eqiad.wmnet mw1300.eqiad.wmnet |
[production] |
| 22:42 |
<krinkle@tin> |
Finished deploy [jobrunner/jobrunner@5f6099f]: (no justification provided) (duration: 08m 18s) |
[production] |
| 22:34 |
<krinkle@tin> |
Started deploy [jobrunner/jobrunner@5f6099f]: (no justification provided) |
[production] |
| 22:02 |
<krinkle@tin> |
Finished deploy [jobrunner/jobrunner@5f6099f]: (no justification provided) (duration: 07m 58s) |
[production] |
| 21:54 |
<krinkle@tin> |
Started deploy [jobrunner/jobrunner@5f6099f]: (no justification provided) |
[production] |
| 21:43 |
<Krinkle> |
Attempt to deploy mediawiki/services/jobrunner – https://gerrit.wikimedia.org/r/#/c/349364/ - failed. |
[production] |
| 19:56 |
<dzahn@neodymium> |
conftool action : set/pooled=yes; selector: name=mw2202.codfw.wmnet |
[production] |
| 19:48 |
<robh> |
starting wipe on cp400[1-4] per T169020 |
[production] |
| 19:15 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.10 |
[production] |
| 18:59 |
<demon@tin> |
Synchronized php-1.30.0-wmf.9/extensions/MobileFrontend/extension.json: One (more) last thing (duration: 02m 49s) |
[production] |
| 18:51 |
<demon@tin> |
Synchronized php-1.30.0-wmf.9/extensions/MobileFrontend/extension.json: One last thing (duration: 02m 55s) |
[production] |
| 18:42 |
<mutante> |
netmon1002 - reinstall OS - didn't use the right partman recipe - didn't have md0 - revoke old puppet cert , salt-key, scheduled downtime, services over at netmon2001 |
[production] |
| 18:36 |
<mutante> |
mw2202 - scheduled downtime - mainboard replacement |
[production] |
| 18:36 |
<ejegg> |
updated payments-wiki from bdc52265d78c55cfc6a732f14519f5f79c9d2d94 to c3be2bfd8f2b9f9eac4c80b45096713c7fdcceff |
[production] |
| 18:29 |
<demon@tin> |
Finished scap: mobilefrontend wmf.9 + forced l10n rebuild (duration: 20m 53s) |
[production] |
| 18:26 |
<mutante> |
mw2202 - remove /etc/udev/rules.d/70-persistent-net.rules for mainboard replacement - to detect new NICs with new MACs (T170307) |
[production] |
| 18:24 |
<dzahn@neodymium> |
conftool action : set/pooled=no; selector: name=mw2202.codfw.wmnet |
[production] |
| 18:08 |
<demon@tin> |
Started scap: mobilefrontend wmf.9 + forced l10n rebuild |
[production] |
| 18:02 |
<ottomata> |
stopping kafka on kafka1012 again, i think we swapped the wrong disk T168927 |
[production] |
| 17:55 |
<awight@tin> |
Finished deploy [ores/deploy@1d35aa5]: T170485 (duration: 35m 06s) |
[production] |
| 17:47 |
<mutante> |
smokeping - switched to netmon2001 - ping times to codfw hosts went down - ping times to eqiad hosts went up - since service is on both but data has been synced over |
[production] |
| 17:41 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: labtest typofix for tgr (duration: 00m 46s) |
[production] |
| 17:21 |
<mobrovac@tin> |
Finished deploy [parsoid/deploy@1eaa07e]: Bring wtp2019 up to date and repool it - T146113 (duration: 01m 02s) |
[production] |
| 17:20 |
<mobrovac@tin> |
Started deploy [parsoid/deploy@1eaa07e]: Bring wtp2019 up to date and repool it - T146113 |
[production] |
| 17:20 |
<awight@tin> |
Started deploy [ores/deploy@1d35aa5]: T170485 |
[production] |
| 17:18 |
<demon@tin> |
Finished scap: testwiki to wmf.10 + l10n cache build (duration: 24m 23s) |
[production] |
| 17:16 |
<ottomata> |
stopping kafka broker on kafka1012 to replace disk T168927 |
[production] |
| 16:53 |
<demon@tin> |
Started scap: testwiki to wmf.10 + l10n cache build |
[production] |