|
2014-09-09
§
|
| 11:29 |
<_joe_> |
git.wikimedia.org works now, no action needed |
[production] |
| 11:26 |
<MatmaRex> |
git.wikimedia.org is down: Error: 503, Service Unavailable |
[production] |
| 10:04 |
<_joe_> |
also re-enabling puppet |
[production] |
| 10:03 |
<_joe_> |
restarting manually apache on mw1178,mw1192,mw1163,mw1130,mw1018 as they started with the wrong pidfile before my fix |
[production] |
| 09:24 |
<_joe_> |
disabling puppet on appservers |
[production] |
| 08:55 |
<godog> |
launched "iptables" on tin to check current rules and it loaded iptables modules, logging for future reference |
[production] |
| 08:10 |
<_joe_> |
re-enabling puppet on appservers and imagescalers, change is good |
[production] |
| 08:08 |
<_joe_> |
restarted apache2 on mw1018 |
[production] |
| 08:07 |
<_joe_> |
stopping apache on mw1018 for inspection |
[production] |
| 07:37 |
<_joe_> |
that was on appservers |
[production] |
| 07:36 |
<_joe_> |
disabling puppet, releasing a potentially harmful apache change |
[production] |
| 04:56 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Sep 9 04:56:25 UTC 2014 (duration 56m 24s) |
[production] |
| 03:44 |
<LocalisationUpdate> |
completed (1.24wmf20) at 2014-09-09 03:44:07+00:00 |
[production] |
| 03:11 |
<LocalisationUpdate> |
completed (1.24wmf19) at 2014-09-09 03:11:27+00:00 |
[production] |
| 02:38 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-09-09 02:38:38+00:00 |
[production] |
| 01:02 |
<ebernhardson> |
Synchronized php-1.24wmf20/extensions/Flow/includes/Content/BoardContentHandler.php: Sync BoardContentHandler.php for Flow in 1.24wmf20 (duration: 00m 04s) |
[production] |
| 00:22 |
<mutante> |
re-enabled mw1070 in pybal |
[production] |
| 00:19 |
<ebernhardson> |
Finished scap: Repeat SWAT scap deployment due to possible sync-common failure (duration: 38m 50s) |
[production] |
|
2014-09-08
§
|
| 23:59 |
<ori> |
restarted rsync on mw1070 to unblock scap |
[production] |
| 23:40 |
<ebernhardson> |
Started scap: Repeat SWAT scap deployment due to possible sync-common failure |
[production] |
| 23:39 |
<ebernhardson> |
Finished scap: SWAT deploy updates to Flow, Echo and Thanks (duration: 24m 00s) |
[production] |
| 23:34 |
<mutante> |
disabled mw1070 in pybal because it refused sync |
[production] |
| 23:31 |
<ebernhardson> |
scap failed to connect to mw1070. Repeated message: rsync: failed to connect to mw1070.eqiad.wmnet (10.64.16.50): Connection refused (111) |
[production] |
| 23:15 |
<ebernhardson> |
Started scap: SWAT deploy updates to Flow, Echo and Thanks |
[production] |
| 23:03 |
<ebernhardson> |
Synchronized wmf-config/InitialiseSettings.php: gerrit:159089 Enable $wgContentHandlerUseDB on mediawikiwiki, testwiki, & test2wiki (duration: 00m 05s) |
[production] |
| 20:14 |
<subbu> |
deployed Parsoid ce108cb5 |
[production] |
| 18:01 |
<demon> |
Synchronized php-1.24wmf19/extensions/Wikidata: Updating Wikidata to f1d2110 (duration: 00m 09s) |
[production] |
| 17:19 |
<mutante> |
disabled notifications for puppet freshness on neon |
[production] |
| 16:19 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: svwiki: Cirrus as primary (duration: 00m 04s) |
[production] |
| 15:42 |
<manybubbles> |
Synchronized php-1.24wmf20/extensions/Wikidata/: SWAT update wikidata to fix add links widget (duration: 00m 06s) |
[production] |
| 15:32 |
<manybubbles> |
Synchronized php-1.24wmf20/extensions/LiquidThreads/: SWAT update liquidthreads to fix some missing images (duration: 00m 04s) |
[production] |
| 15:28 |
<manybubbles> |
15:13:53 Synchronized php-1.24wmf19/extensions/WikiLove/: SWAT fix for WikiLove (duration: 00m 04s) |
[production] |
| 15:28 |
<manybubbles> |
this is the missing log: |
[production] |
| 15:27 |
<manybubbles> |
sync logging was down so it missed some syncing I just did. |
[production] |
| 15:25 |
<manybubbles> |
Synchronized php-1.24wmf20/extensions/WikiLove/: (no message) (duration: 00m 05s) |
[production] |
| 15:20 |
<manybubbles> |
Synchronized wmf-config: SWAT another cirrus setting update (duration: 00m 04s) |
[production] |
| 15:10 |
<manybubbles> |
Synchronized wmf-config: SWAT finish updating Cirrus settings (duration: 00m 05s) |
[production] |
| 15:10 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT update some cirrus settings (duration: 00m 04s) |
[production] |
| 15:10 |
<cmjohnson1> |
shutting down neon for memory upgrade |
[production] |
| 14:41 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1072 (duration: 00m 09s) |
[production] |
| 12:23 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1073, depool db1072 (duration: 00m 06s) |
[production] |
| 11:07 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1073wq (duration: 00m 09s) |
[production] |
| 10:55 |
<_joe_> |
re-enabled puppet, the change results in a no-op as expected |
[production] |
| 10:42 |
<_joe_> |
disabling puppet on all appservers while updating apache config. |
[production] |
| 04:58 |
<springle> |
Synchronized wmf-config/db-eqiad.php: move enwiki api traffic to db1051/db1066 (duration: 00m 09s) |
[production] |
| 03:37 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Sep 8 03:36:13 UTC 2014 (duration 36m 12s) |
[production] |
| 02:45 |
<LocalisationUpdate> |
completed (1.24wmf20) at 2014-09-08 02:44:47+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf19) at 2014-09-08 02:32:37+00:00 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-09-08 02:19:52+00:00 |
[production] |