Wikimedia Bugzilla is closed!

Are those "..."s normal updates that worked or just the same exception repeated over and over and over again?

Also, what does Elasticsearch's health check (ES_HOST:9200/_cluster/health?pretty) say?

If it says RED then the problem is that elasticsearch isn't happy.  I'm happy to make the error message less obtuse and more helpful.

the "..." are just the other 494 lines that said: Timeout waiting for [1m]

Sorry, I do not know how to use this: "ES_HOST:9200/_cluster/health?pretty"
But with elasticsearch-head I get this (yellow)

{

    cluster_name: kc-alpha1
    status: yellow
    timed_out: false
    number_of_nodes: 1
    number_of_data_nodes: 1
    active_primary_shards: 8
    active_shards: 8
    relocating_shards: 0
    initializing_shards: 0
    unassigned_shards: 16

}

Just for you info. It worked a couple of weeks ago and I did not change my elasticsearch config. But you never know....

I've got it!

A few weeks ago we changed the default number of replicas from 1 to 2 so we could have more redundancy during updates.  Which is cool and all, but when you have a single node elasticsearch you can't have _any_replicas.

The reason it was working before is that we are using elasticsearch's default write consistency: quorum.  My guess is quorum let the write through when the master was up but the other shards weren't.

You can work around this by adding this to your LocalSettings:
$wgCirrusSearchContentReplicaCount = array( 'content' => 0, 'general' => 0 );


We're going to make that the default and add a note to the readme about changing it for production use.  That commit will go in later on today.

I'm going to usurp this bug to make the error message better.

This is the LocalSettings.php now:

##Start --------------------------------------- CirrusSearch
require_once( "$IP/extensions/Elastica/Elastica.php" );
require_once( "$IP/extensions/CirrusSearch/CirrusSearch.php" );
#$wgDisableSearchUpdate = true;
$wgCirrusSearchServers = array( 'localhost' );
$wgCirrusSearchContentReplicaCount = array( 'content' => 0, 'general' => 0 );
#$wgSearchType = 'CirrusSearch';
##End   --------------------------------------- CirrusSearch

Input:
 php forceSearchIndex.php

Output:
...
...
...
index: /wikidb_mw1_test_general_first/page/499 caused UnavailableShardsException[[wikidb_mw1_test_general_first][3] [3] shardIt, [1] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@6a3b300c]

Indexed 475 pages ending at 499 at 4/second
Indexed a total of 475 pages at 4/second

Sorry, for replica count to take effect you have to rerun updateSearchIndexConfig.  I'll make sure variables like that are more clearly marked in future.

now it only took 10 seconds..... :-)
you are the BEST!!!

php forceSearchIndex.php

Indexed 475 pages ending at 499 at 106/second
Indexed a total of 475 pages at 106/second