|
2017-01-27
§
|
| 10:35 |
<paravoid> |
manually running certspotter -all_time as my user on einstenium (will take a few days to complete) |
[production] |
| 10:21 |
<legoktm> |
added addshore to labs-tools-wikibugs2 gerrit group |
[production] |
| 08:13 |
<moritzm> |
uploaded openssl 1.1.0d packages for jessie-wikimedia to carbon |
[production] |
| 02:58 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Jan 27 02:58:52 UTC 2017 (duration 5m 46s) |
[production] |
| 02:53 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.9) (duration: 14m 14s) |
[production] |
| 02:21 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.8) (duration: 07m 57s) |
[production] |
| 01:20 |
<twentyafterfour> |
deploying hotfix for phabricator refs T154479 |
[production] |
| 00:48 |
<mutante> |
carbon - moved the 1.5TB /srv/"mirrors.off", which used to be mirrors but is now on sodium, into / to that /srv/ can be synced without this |
[production] |
| 00:32 |
<dereckson@tin> |
Synchronized wmf-config/throttle.php: Fix throttle rule for Her Girl Friday + Lenny Unconference (T156278) (duration: 00m 53s) |
[production] |
| 00:12 |
<volans> |
re-enabled puppet (with a temporary fix to keep parsoid-vd and parsoid-vd-client stopped) on ruthenium T156177 |
[production] |
|
2017-01-26
§
|
| 23:49 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.29.0-wmf.8 |
[production] |
| 23:37 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.9 |
[production] |
| 23:33 |
<robh> |
archiva.w.o maint done, uses new LE cert. |
[production] |
| 23:14 |
<robh> |
going to try to convert archiva.wikimedia.org from GS to LE cert. will require rehup of nginx |
[production] |
| 22:58 |
<mutante> |
db1019, db1042 - revoke puppet certs, delete salt keys, schedule icinga downtime, stop services (T149793, T146265) |
[production] |
| 22:56 |
<mutante> |
analytics1015, analytics1026 - puppet node clean (again?) - again having problems to remove decom'ed nodes from Icinga (T147313) |
[production] |
| 22:53 |
<mutante> |
analytics1015, analytics1026 - puppet node clean (again?) - again having problems to remove decom'ed nodes from Icinga |
[production] |
| 22:46 |
<mutante> |
mw1181, mw1272, mw1212, mw1174 - service hhvm restart |
[production] |
| 22:06 |
<mobrovac@tin> |
Finished deploy [trending-edits/deploy@e0e32bb]: Bump replay time to 6h for T156411 (duration: 01m 42s) |
[production] |
| 22:04 |
<mobrovac@tin> |
Starting deploy [trending-edits/deploy@e0e32bb]: Bump replay time to 6h for T156411 |
[production] |
| 20:30 |
<andrewbogott> |
refreshing logins on wikitech |
[production] |
| 19:13 |
<elukey> |
restore analytics1001 as RM and HDFS masters |
[production] |
| 18:56 |
<otto@tin> |
Finished deploy [eventstreams/deploy@f1a1866]: (no message) (duration: 03m 16s) |
[production] |
| 18:52 |
<otto@tin> |
Starting deploy [eventstreams/deploy@f1a1866]: (no message) |
[production] |
| 18:36 |
<elukey> |
restarting Yarn node managers on an102[89] and an103[01], impacted by the switch restart |
[production] |
| 18:32 |
<paravoid> |
starting pybal on lvs1001/lvs1002/lvs1003 |
[production] |
| 18:15 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1055, 56, 57, 59 (duration: 00m 54s) |
[production] |
| 18:14 |
<paravoid> |
rebooting newly provisioned asw-c2-eqiad to enable mixed mode |
[production] |
| 17:57 |
<elukey> |
boostrapping aqs1007-a cassandra instance |
[production] |
| 17:51 |
<paravoid> |
replacing asw-c2-eqiad |
[production] |
| 17:46 |
<paravoid> |
stopping pybal on lvs1001/lvs1002/lvs1003 |
[production] |
| 17:34 |
<elukey@tin> |
Finished deploy [analytics/aqs/deploy@5917fd4]: (no message) (duration: 02m 25s) |
[production] |
| 17:31 |
<elukey@tin> |
Starting deploy [analytics/aqs/deploy@5917fd4]: (no message) |
[production] |
| 15:43 |
<bblack> |
cache_misc puppet re-enabled and up to date |
[production] |
| 15:37 |
<godog> |
bounce uwsgi on graphite1003 with less workers - T155872 |
[production] |
| 15:35 |
<moritzm> |
installing gnupg2 updates from jessie point update |
[production] |
| 15:15 |
<akosiaris> |
T156242 add /dev/sdb partitions to mdadm devices |
[production] |
| 15:15 |
<bblack> |
puppet disabled on cache_misc for merging complicated stuff |
[production] |
| 15:06 |
<moritzm> |
installing gnupg updates from jessie point update |
[production] |
| 14:52 |
<jynus> |
stopping mysql on db1048 T156373 |
[production] |
| 14:29 |
<zeljkof> |
finished with eu swat |
[production] |
| 14:28 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:334134|IP Cap Lift for Edit-a-Thon (T156258)]] [[gerrit:334156|[throttle] Her Girl Friday + Lenny Unconference / Editathon in NYC, 2017-01-28 (T156278)]] (duration: 00m 41s) |
[production] |
| 13:54 |
<godog> |
delete labs 'instances' graphite three for data >30d, graphite low on disk space |
[production] |
| 13:53 |
<elukey> |
restarting cassandra on aqs100[56] to complete the openjdk update |
[production] |
| 13:32 |
<moritzm> |
rolling restart of maps cluster in eqiad |
[production] |
| 13:08 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1054 - T156225 (duration: 00m 40s) |
[production] |
| 13:05 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.29.0-wmf.9 T156310 |
[production] |
| 13:02 |
<godog> |
reboot ms-be1013 |
[production] |
| 12:54 |
<elukey> |
restarting the aqs1004-b casandra instance to pick up the new openjdk (last test before complete rollout) |
[production] |
| 12:28 |
<elukey> |
restarting the aqs1004-a casandra instance to pick up the new openjdk |
[production] |