|
2018-06-06
ยง
|
| 18:54 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905 |
[production] |
| 18:47 |
<bd808> |
Restarted webservice, stuck in CrashLoopBackOff |
[tools.wmf-task-samtar] |
| 18:43 |
<bd808> |
Webservice in CrashLoopBackOff due to failure to reach video-redis.video.eqiad.wmflabs:6379 |
[tools.video2commons-socketio] |
| 18:38 |
<andrewbogott> |
rebooting labvirt1019, 1020 |
[production] |
| 18:37 |
<bd808> |
Stopped webservice. App trying to run django, but failing to load MySQLdb module. This causes an infinite start, crash, restart loop |
[tools.anni-me] |
| 18:37 |
<Reedy> |
qdel current jobs |
[tools.wikibugs] |
| 18:36 |
<andrewbogott> |
rebooting labvirt1018, 1021, 1022 |
[production] |
| 18:36 |
<bd808> |
Restarted webservice, 25720 CrashLoopBackOff retries logged |
[tools.anni-me] |
| 18:29 |
<andrewbogott> |
rebooting labvirt1017 |
[production] |
| 18:23 |
<andrewbogott> |
rebooting labvirt1016 |
[production] |
| 18:21 |
<andrewbogott> |
rebooting labvirt1015 |
[production] |
| 18:19 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905 |
[production] |
| 18:15 |
<bd808> |
Restarting webservice, pod in CrashLoopBackOff state |
[tools.replag] |
| 18:14 |
<andrewbogott> |
rebooting labvirt1013 |
[production] |
| 18:13 |
<bd808> |
Restarting webservice (webservice --backend kubernetes php5.6 start) |
[tools.admin] |
| 18:11 |
<bd808> |
Pod in CrashLoopBackOff, invesitgating |
[tools.admin] |
| 18:06 |
<andrewbogott> |
rebooting labvirt1012 |
[production] |
| 17:59 |
<andrewbogott> |
rebooting labvirt1011 |
[production] |
| 17:59 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905 |
[production] |
| 17:56 |
<andrewbogott> |
switching primary proxy to novaproxy-01 |
[project-proxy] |
| 17:44 |
<andrewbogott> |
rebooting labvirt1010 |
[production] |
| 17:41 |
<andrewbogott> |
switching primary proxy to novaproxy-02 |
[project-proxy] |
| 17:09 |
<hashar> |
github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow | T186969 |
[releng] |
| 17:02 |
<hashar> |
Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ | T196509 |
[releng] |
| 17:01 |
<hashar> |
github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS | T196509 |
[releng] |
| 17:01 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905 |
[production] |
| 16:59 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1084 fully (duration: 00m 56s) |
[production] |
| 16:51 |
<andrewbogott> |
rebooting labvirt1008 |
[production] |
| 16:33 |
<andrewbogott> |
rebooting labvirt1007 |
[production] |
| 16:25 |
<jynus> |
stop mysql @ db1051 in preparation for decom |
[production] |
| 16:25 |
<andrewbogott> |
rebooting labvirt1006 |
[production] |
| 16:16 |
<andrewbogott> |
rebooting labvirt1005 |
[production] |
| 16:09 |
<andrewbogott> |
rebooting labvirt1004 |
[production] |
| 16:05 |
<XioNoX> |
lvs1002 repooled |
[production] |
| 16:04 |
<andrewbogott> |
rebooting labvirt1002 |
[production] |
| 15:57 |
<twentyafterfour> |
reloading apache on phab1001 to free up some resources |
[production] |
| 15:56 |
<andrewbogott> |
rebooting labvirt1014 |
[production] |
| 15:54 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905 |
[production] |
| 15:51 |
<XioNoX> |
disable pybal on lvs1002 - T187962 |
[production] |
| 15:39 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies (duration: 00m 40s) |
[production] |
| 15:38 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies |
[production] |
| 15:32 |
<_joe_> |
cross enabling videoscalers,jobrunners in their respective pools |
[production] |
| 15:27 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905 |
[production] |
| 15:26 |
<XioNoX> |
stop pybal on lvs1001 |
[production] |
| 15:26 |
<awight@deploy1001> |
Finished deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 (duration: 25m 37s) |
[production] |
| 15:13 |
<_joe_> |
adding jobrunners, videoscalers to both pools with equal weight in codfw |
[production] |
| 15:06 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=videoscaler,dc=codfw,service=nginx |
[production] |
| 15:06 |
<urandom> |
upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905 |
[production] |
| 15:01 |
<awight@deploy1001> |
Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 |
[production] |
| 14:39 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw22(4[1-5]|5[3-8]|6[1-9]).* |
[production] |