Slave Replication issue

All installation and configuration problems and questions

Moderators: gerski, enjay, williamconley, Op3r, Staydog, gardo, mflorell, MJCoate, mcargile, Kumba, Michael_N

Slave Replication issue

Postby frequency » Mon Jul 09, 2018 5:24 pm

HI,

This may be a off-topic a bit. We are replicating vicidial database using normal vicibox installation method. The cluster has about a hundred agents. Slave is using LSI MegaRAID Level 0 Crucial MX500 SSD's. The server is on same network with <1ms latency. once more than 40 agents are logged on the dialer, the slave DB starts to show i/o wait time of 10-12% constant at 6000-9000kB/s writes and slave behind master increased each second.

I have tried replacing the raid card and also the ssd's with fresh installation but the issue persists. the temps are normal. We have another cluster with 100 agents with reporting slave server working flawlessly with similar equipment. what could be the issue?

TIA
frequency
 
Posts: 117
Joined: Mon Jun 13, 2016 11:18 am

Re: Slave Replication issue

Postby Vince-0 » Tue Jul 10, 2018 10:55 am

Slave DB is single threaded as far as I know. You may be able to tweak slave config/mariadb versions to help with this.
Otherwise maybe raid write cache needs to be enabled?

Is your "iostat -x 1" util% at 99% ?
Vince-0
 
Posts: 272
Joined: Fri Mar 02, 2012 4:27 pm
Location: South Africa

Re: Slave Replication issue

Postby frequency » Tue Jul 10, 2018 11:46 am

%util is hovering between 89% to 100%.

MariaDB is updated to latest version via zypper up command.

I'll check how to turn on the raid write cache on lsi card

EDIT: enabling write-cache over the megaraid resolved the issue.

Thanks a lot lot!
frequency
 
Posts: 117
Joined: Mon Jun 13, 2016 11:18 am

Re: Slave Replication issue

Postby frequency » Wed Jul 11, 2018 4:25 pm

The server seems to be retrieving "Agent status report" for a single day in 90 seconds now...User stats seems to be opening just fine.
frequency
 
Posts: 117
Joined: Mon Jun 13, 2016 11:18 am

Re: Slave Replication issue

Postby williamconley » Wed Jul 11, 2018 5:10 pm

We've also found multi-threading and "conservative" parallel mode to help a "not-so-powerful" slave keep up with a more powerful master server.

Code: Select all
Parallel_Mode => conservative
slave_parallel_threads=256
slave_domain_parallel_threads=256
thread_concurrency=512


In this case, a 40-core master and a 4 core slave.
Vicidial Installation and Repair, plus Hosting and Colocation
Newest Product: Vicidial Agent Only Beep - Beta
http://www.PoundTeam.com # 352-269-0000 # +44(203) 769-2294
williamconley
 
Posts: 20018
Joined: Wed Oct 31, 2007 4:17 pm
Location: Davenport, FL (By Disney!)

Re: Slave Replication issue

Postby frequency » Thu Jul 12, 2018 11:43 am

williamconley wrote:We've also found multi-threading and "conservative" parallel mode to help a "not-so-powerful" slave keep up with a more powerful master server.

Code: Select all
Parallel_Mode => conservative
slave_parallel_threads=256
slave_domain_parallel_threads=256
thread_concurrency=512


In this case, a 40-core master and a 4 core slave.


Seems flawless now. Thank you..
frequency
 
Posts: 117
Joined: Mon Jun 13, 2016 11:18 am


Return to Support

Who is online

Users browsing this forum: Bing [Bot] and 73 guests