Top 6 Factors Impacting Your Data Replication Performance

This is an article about Zerto, the award-winning IT Resilience platform I was first exposed to in 2011 and loved so much that I joined the team in 2012. For help with any Zerto-specific terminology, see Zerto’s official product documentation. This post refers to Zerto Virtual Replication versions 6.x and earlier.

Whenever I’m speaking with a Zerto customer or partner about IT Resilience and things get technical, we’ll inevitably come to the part where someone asks how fast their data will go from point A to point B. While this may not seem immediately meaningful to the business, the underlying contexts are minimizing risk, typically measured as data loss, and increasing revenues, typically measured as reducing the time taken to return to production.

Each time, a decision maker on the technical team will go right to the volume of data they need to move from point A to point B and their ISP’s listed internet connection speed. “This application is backed by a 46 TB database”, “this application has 8 virtual machines (VMs) with 1 TB of volumes each”, and so on, coupled with, “we have an X Gbps connection so we expect this to go at X Gbps.”

It’s at this time we get to have some straight-talk.

Here’s the reality: getting data out of one spot in a data center, and into another spot in another data center, has maybe 50% to do with your provider’s internet connection speed. There’s a pile of equipment, software, and configurations sitting between where your data lives and the point that I call “just getting your data out the door.” There’s another list of stuff involved in getting your data across the expanse of time and space leading to the next building. Then, that data has to get inside the next building and ultimately it’s got to land on your target storage.

Each of the factors below can interact with others in ways that are often unknown to the customer’s IT staff and the ISP’s engineers themselves. The good news is that these factors are always in play. I say “good news” because that means you can always be identifying them, keeping track of them, and so on.

  • Network devices: your routers, switches, gateways, load balancers, and then all of the ISP’s gear…
  • Network services: your MPLS, VPN, QoS, bandwidth shaping, and then all of your ISP’s services configurations…
  • Network overheads and limits: hypervisor offload, frame rate limits, packet-per-second limits, max-concurrent-connection limits, latency, end-to-end MTU configurations, other application workloads…
  • Security devices: firewalls, any host security hardware such as data encryption modules or encrypted NICs
  • Security services: Intrusion Detection, Intrusion Prevention, other network filtering services; any host security software such as host anti-virus
  • Storage devices: read/write split, RAID type, caching, backplane performance, compression, deduplication, latency, other application workloads

I recall an issue where a customer had a single application in Production that was running at over 20,000 IOPS and they were getting upset with our Support team because they weren’t getting the replication performance they expected.

…what they neglected to say (and what we almost immediately saw in our log files) was their target storage array’s write performance wasn’t anywhere near capable of keeping up with that speed. I’m not an “I told you so” person, but I did make sure to go over the above list with them for a second time.

I’ve seen performance held up by storage factors far more frequently than other factors.

In my experience, most businesses put all their money into “production” and peanuts into recovery, including the recovery site hardware. As older storage arrays continue to be replaced by modern equipment, and as IT resilience continues moving up the list of business priorities, I believe we’ll see this change for the better.

As an aside, I’ve been in so many meetings and conference calls where fingers are pointed left, right, and sideways because nobody wants to believe their piece of the puzzle is the problem. One of the best parts about IT, though, is that it’s easy to cut through the noise and get straight to the facts.

Stop Pitching Products and Features

A couple of co-workers from my company’s Sales Development team asked me what they should be talking about when they reach out to a potential customer to try and get a meeting. I asked about their current talk track and listened as they told me about the latest feature. I winced at hearing about the bells and whistles, the knobs and the dials.

They were a bit stunned when I said, “I wouldn’t talk about any of that.”

This is a team with some pretty amazing stats in terms of booking meetings with potential customers. Still, I’d listened in on some calls and thought I might be able to offer a technique to increase their success rate.

“If it’s me, I’m not talking about the product at all. Follow the script, for now, and we’ll work on it over time. Just know I wouldn’t be talking at all about the product.”

One of them asked, “Really?”

“Really. I’d call, introduce myself, and launch into how I’m calling to see if you’ve got 20 minutes in the next week or so to meet with Jim Bob Sales Rep to discuss the 3 biggest challenges their peers are facing today. No products, no slides, Jim will just share some of the major pains your peers are facing. If you’re facing even 1 of them, like some of our customers were before signing up, then Jim can set a follow-up meeting to discuss our solution. Then I’d ask for a day and time. Like, ‘How’s Thursday at 2pm look for you?’ Don’t ask for a meeting, just assume there’s going to be one.”

Another moment passed by as they contemplated the strategy.

“What if I don’t know the customer’s problem?” one of them replied.

“What do you mean you d… of course you know their problems!” I shot back. “We hear the same problems from our customers all the time! The same problems are why they buy. What do you hear in healthcare? Write it down. What do you hear in finance? Write it down. Same for everyone. Put it all down. There’s your cheat sheet for the next customer in the same industry. In any industry.”

“Look, I was a customer once, right? So pretend I’m your potential customer. I’m busy solving a hundred other problems. Why would I talk to you if you have nothing to offer? You’re asking for 5 or 10 minutes to bombard me with questions about my current approach or just pitch some features? I’d hang up on you.”

They nodded in agreement.

“Shouldn’t you just assume that since I’m not yet a customer I must have the same or similar problems as others in the same industry have had?”

And just like that, lightbulbs.

Your customer has a pain point that your product addresses. You know what those pain points are and how to overcome them because you’ve done it dozens or hundreds of times. Don’t sell products. Don’t sell features. Don’t pitch bells and whistles. Sell on being knowledgeable in your customer’s industry and having experience with overcoming those problems.

Climbing the mountain

When I joined Zerto back in 2012 I knew I was in for an adventure. At the time I was hired the company had less than 50 employees worldwide, a few dozen reseller-partners, a handful of Cloud Providers offering Zerto-as-a-Service, and a 2.0 product which usually took a couple of hours of hands-on work to get up and running. I had seen Zerto in a lab but never in real production use. The company had about 100 customers with what felt like good product/market fit, a recent Series-B, and who I felt (and still do) was the right team for the job.

The next two weeks are a blur of me sitting in the cube of the other Sales Engineer at the time, learning the Zerto software and creating process (read: making it up as I went) out of what is better known as typical startup chaos. Case in point: due to a massive workload and a need to get going ASAP I downloaded all of the documentation on Zerto to an iPad that I took with me everywhere. The morning train commute, the bathroom, when I say “everywhere” I mean “everywhere”! 10 days later I was leading my first technical calls.

Coming from a reseller that I thought was the epitome of “high activity”, I have to admit that I was floored. Potential customers and partners were exploding with interest. I’d hop on a plane with one of my sales reps, hit 7 meetings a day through a half-dozen states in a week, and fly home for a weekend of rest and working in my lab. Come Monday I’d have ten conference calls a day through 8 or 9pm that Friday. Rinse and repeat. Late in the evening on December 31st, 2012, with the last order in the system for processing, the reports showed that it all paid off: we all saw that we had brought on over 250 customers through the year, or at least one new customer every day since I started.

Now fast-forward several years.

We recently held an Americas business review. These are standard in sales and different companies hold them at different frequencies through the year. A sales manager asked a question regarding one of our key performance indicators, or, “How many meetings should we hold our folks accountable to each week?” A few opinions shot up around the room and the team began a healthy debate. I was standing in the back of the room and, as things wound down, flashed both hands out several times to our VP of Sales who noticed and said, “Sean’s giving a pretty big number back there!” I spoke up:

“When I was a Sales Engineer I wouldn’t get on a plane for less than 7 meetings a day. So 25 or 30 a week sounds about right.” I got some nods as well as some eye rolls, like I had just said, “When I was your age…”

In hindsight, I could have articulated my vision more clearly at the time.

When you work for a startup you aren’t just working for a company but really building it. You’re creating something from nearly nothing. Zerto has thousands of customers and hundreds of partners and yet we truly are at the bottom of a mountain. VMware, the leader in server virtualization today with the vSphere product family, boasts over 500,000 customers globally. Zerto software also supports Microsoft’s server virtualization platform, Hyper-V. We’ve extended support for disaster recovery to Amazon Web Services and Microsoft Azure, and we’re currently working on ways to get virtual machines back out of those two public clouds. In fact, “bottom of the mountain” doesn’t quite describe it.

It’s more like this:

Mount Everest seemed like an appropriate metaphor

Since the business review a few Sales Managers and SE Managers have asked exactly what I meant that day. This has been and continues to be my answer: At Zerto we have an enormous mountain of opportunity ahead of us. VMware alone has over 500,000 customers. Remind me again of how many we have? It is not a small number, but it is nowhere near five-hundred thousand. It’s not 100,000. We aren’t even 20% of the way there. And that’s just one platform! We’re cross-platform, baby! We aren’t yet climbing the mountain, we’re on the long and glorious trek to the base of it. Have breakfast with a partner. Have lunch with a customer. Take a potential customer to dinner. That’s three meetings right there. That is three steps closer to the base of the mountain. This is the warm-up. Stretch your legs, take in the view, you’re in for a wonderful adventure.

Every day hundreds of Zertonians around the world wake up with one goal: create a world of uninterrupted technology for businesses & people. Think you have what it takes? We’re hiring hundreds of people this year across dozens of roles. Come join us on our amazing journey.